1. Forum
  3. Application
  5. Troubleshooting
  7. [SOLVED] Bugs in generated HLSL code (2.0.3f1)

[SOLVED] Bugs in generated HLSL code (2.0.3f1)

  • Posts: 343
4 years 2 months ago - 4 years 2 months ago #1 by OtisInf
Hi,

I think I ran into several issues.

First:
texture Otis_FogColorFromMouseTarget { Width=1; Height=1; Format= Otis_RENDERMODE;};
sampler2D Otis_FogColorFromMouseSampler { Texture = Otis_FogColorFromMouseTarget;};

//...
float4 fogColor = tex2Dfetch(Otis_FogColorFromMouseSampler, int2(0, 0));

gives on the tex2Dfetch line:
error X3017: '__tex2Dfetch': cannot implicitly convert from 'const int2' to 'int4'
Using tex2D() works (with float coords)

Then:
What I want is this:
float4 PS_Otis_AFG_PerformGetFogColorFromFrameBuffer(float4 vpos: SV_Position, float2 texcoord: TEXCOORD) : SV_Target
{
	if(Otis_MouseToggleKeyDown)
	{
		return tex2D(ReShade::BackBuffer, ReShade::MouseCoords * ReShade::PixelSize);
	}
	else
	{
		discard;
		return float4(AFG_Color, 1.0); // should be ignored anyway.
	}
}

// with: 
pass Otis_AFG_PassFogColorFromMouse
{
	VertexShader = ReShade::VS_PostProcess;
	PixelShader = PS_Otis_AFG_PerformGetFogColorFromFrameBuffer;
	RenderTarget = Otis_FogColorFromMouseTarget;
}
So I want to NOT write to the render target if the key isn't ON, so discard the pixel. However discard here doesn't discard the pixel, it will write black (0,0,0,0) to the texture as if it's re-initialized each time.

So I thought, maybe it's me being stupid, so I'll just use clip(-1); but when I use clip(-1) like:
		clip(-1);   // line 80

I get:
AdaptiveFog.fx(80, 1): error X3004: undeclared identifier 'clip'
AdaptiveFog.fx(80, 9): error X3000: syntax error: unexpected ';'
not sure, but it looks like 'clip' isn't a known function? I'm using Dx11, on a 780, which IMHO is way inside the required shader model.

(edit) I see 'clip' isn't in the list of known intrinsic functions for reshade, is this because there's no equivalent under OpenGL? I can live without using clip, I just want to discard the complete rendering of the pixel so it will keep the previous value written to the texture. I recon this isn't possible?

The whole idea is that in a pixel shader pass ran after the pass specified above I read from the fog color sampler, which contains the last color on the mouse coordinates when the toggle key was pressed. This way I can pick a color from the scene and keep it after moving the mouse. However when I toggle the key off again, I go back to black, so it overwrites the pixel stored with a value, even though discard is used.

Please Log in or Create an account to join the conversation.

  • Posts: 3740
4 years 2 months ago - 4 years 2 months ago #2 by crosire

OtisInf wrote: gives on the tex2Dfetch line:
error X3017: '__tex2Dfetch': cannot implicitly convert from 'const int2' to 'int4'

That's a known issue and fixed already. I had a parameter mismatch of tex2Dfetch between the parser and HLSL code generator. Should be "tex2Dfetch(sampler2D, int4)" (where .w is the mipmap level), but it's "tex2Dfetch(sampler2D, int2)" in 2.0.3.

OtisInf wrote: So I want to NOT write to the render target if the key isn't ON, so discard the pixel. However discard here doesn't discard the pixel, it will write black (0,0,0,0) to the texture as if it's re-initialized each time.

That's because each rendertarget is cleared to zero before it is bound. This was a requirement for SMAA to work.

OtisInf wrote: not sure, but it looks like 'clip' isn't a known function? I'm using Dx11, on a 780, which IMHO is way inside the required shader model.

(edit) I see 'clip' isn't in the list of known intrinsic functions for reshade, is this because there's no equivalent under OpenGL? I can live without using clip, I just want to discard the complete rendering of the pixel so it will keep the previous value written to the texture. I recon this isn't possible?

ReShade FX has no "clip", because the exact same behavior can be achieved with a conditional discard, which is much clearer (it's obvious the shader code won't continue after a discard statement, not so much after a function call). Keep in mind, you don't write HLSL, even though it looks similar.

Please Log in or Create an account to join the conversation.

  • Posts: 343
4 years 2 months ago #3 by OtisInf

crosire wrote:

OtisInf wrote: gives on the tex2Dfetch line:
error X3017: '__tex2Dfetch': cannot implicitly convert from 'const int2' to 'int4'

That's a known issue and fixed already. I had a parameter mismatch of tex2Dfetch between the parser and HLSL code generator. Should be "tex2Dfetch(sampler2D, int4)" (where .w is the mipmap level), but it's "tex2Dfetch(sampler2D, int2)" in 2.0.3.

Ok thanks, will work around this.

OtisInf wrote: So I want to NOT write to the render target if the key isn't ON, so discard the pixel. However discard here doesn't discard the pixel, it will write black (0,0,0,0) to the texture as if it's re-initialized each time.

That's because each rendertarget is cleared to zero before it is bound. This was a requirement for SMAA to work.

I feared as much. However it's then a blur to me how I can preserve a value across frames. In another thread you gave the advice to use a 1x1 texture for this. However if that's cleared before it's bound, I don't know what else to use as the clearing before bound makes it lose the values I wanted to preserve across frames ;) I obviously miss something like a flag to set somewhere to make it not being cleared or something else. Any advice would be great :)

OtisInf wrote: not sure, but it looks like 'clip' isn't a known function? I'm using Dx11, on a 780, which IMHO is way inside the required shader model.

(edit) I see 'clip' isn't in the list of known intrinsic functions for reshade, is this because there's no equivalent under OpenGL? I can live without using clip, I just want to discard the complete rendering of the pixel so it will keep the previous value written to the texture. I recon this isn't possible?

ReShade FX has no "clip", because the exact same behavior can be achieved with a conditional discard, which is much clearer (it's obvious the shader code won't continue after a discard statement, not so much after a function call). Keep in mind, you don't write HLSL, even though it looks similar.

Ah thanks for the explanation. Now if I can get the value to be preserved it would be great...

Please Log in or Create an account to join the conversation.

  • Posts: 3740
4 years 2 months ago - 4 years 2 months ago #4 by crosire

OtisInf wrote: I feared as much. However it's then a blur to me how I can preserve a value across frames. In another thread you gave the advice to use a 1x1 texture for this. However if that's cleared before it's bound, I don't know what else to use as the clearing before bound makes it lose the values I wanted to preserve across frames ;) I obviously miss something like a flag to set somewhere to make it not being cleared or something else. Any advice would be great :)

The ugly workaround solution would be to create a second "backup" texture. So:
  • pass 1 [RenderTarget = texture 2]: copy texture 1 to texture 2
  • pass 2 [RenderTarget = texture 1]: instead of discard, read the data you want to preserve from texture 2
The following user(s) said Thank You: OtisInf

Please Log in or Create an account to join the conversation.

  • Posts: 343
4 years 2 months ago #5 by OtisInf

crosire wrote:

OtisInf wrote: I feared as much. However it's then a blur to me how I can preserve a value across frames. In another thread you gave the advice to use a 1x1 texture for this. However if that's cleared before it's bound, I don't know what else to use as the clearing before bound makes it lose the values I wanted to preserve across frames ;) I obviously miss something like a flag to set somewhere to make it not being cleared or something else. Any advice would be great :)

The ugly workaround solution would be to create a second "backup" texture. So:
  • pass 1 [RenderTarget = texture 2]: copy texture 1 to texture 2
  • pass 2 [RenderTarget = texture 1]: instead of discard, read the data you want to preserve from texture 2


Heh thanks :) Will try it out.

Please Log in or Create an account to join the conversation.

  • Posts: 343
4 years 2 months ago #6 by OtisInf
Works like a charm! :)

Ok, so to recap this thread: problem 1 is already solved in a new build, problem 2 has a workaround, problem 3 (clip) is not supported. Solved :)

Please Log in or Create an account to join the conversation.