1. Forum
  3. ReShade
  5. Troubleshooting
  7. [SOLVED] Bugs in generated HLSL code (2.0.3f1)

[SOLVED] Bugs in generated HLSL code (2.0.3f1)

More
6 years 1 month ago - 6 years 1 month ago #1 by OtisInf
[SOLVED] Bugs in generated HLSL code (2.0.3f1) was created by OtisInf
Hi,

I think I ran into several issues.

First:
texture Otis_FogColorFromMouseTarget { Width=1; Height=1; Format= Otis_RENDERMODE;};
sampler2D Otis_FogColorFromMouseSampler { Texture = Otis_FogColorFromMouseTarget;};

//...
float4 fogColor = tex2Dfetch(Otis_FogColorFromMouseSampler, int2(0, 0));

gives on the tex2Dfetch line:
error X3017: '__tex2Dfetch': cannot implicitly convert from 'const int2' to 'int4'
Using tex2D() works (with float coords)

Then:
What I want is this:
float4 PS_Otis_AFG_PerformGetFogColorFromFrameBuffer(float4 vpos: SV_Position, float2 texcoord: TEXCOORD) : SV_Target
{
	if(Otis_MouseToggleKeyDown)
	{
		return tex2D(ReShade::BackBuffer, ReShade::MouseCoords * ReShade::PixelSize);
	}
	else
	{
		discard;
		return float4(AFG_Color, 1.0); // should be ignored anyway.
	}
}

// with: 
pass Otis_AFG_PassFogColorFromMouse
{
	VertexShader = ReShade::VS_PostProcess;
	PixelShader = PS_Otis_AFG_PerformGetFogColorFromFrameBuffer;
	RenderTarget = Otis_FogColorFromMouseTarget;
}
So I want to NOT write to the render target if the key isn't ON, so discard the pixel. However discard here doesn't discard the pixel, it will write black (0,0,0,0) to the texture as if it's re-initialized each time.

So I thought, maybe it's me being stupid, so I'll just use clip(-1); but when I use clip(-1) like:
		clip(-1);   // line 80

I get:
AdaptiveFog.fx(80, 1): error X3004: undeclared identifier 'clip'
AdaptiveFog.fx(80, 9): error X3000: syntax error: unexpected ';'
not sure, but it looks like 'clip' isn't a known function? I'm using Dx11, on a 780, which IMHO is way inside the required shader model.

(edit) I see 'clip' isn't in the list of known intrinsic functions for reshade, is this because there's no equivalent under OpenGL? I can live without using clip, I just want to discard the complete rendering of the pixel so it will keep the previous value written to the texture. I recon this isn't possible?

The whole idea is that in a pixel shader pass ran after the pass specified above I read from the fog color sampler, which contains the last color on the mouse coordinates when the toggle key was pressed. This way I can pick a color from the scene and keep it after moving the mouse. However when I toggle the key off again, I go back to black, so it overwrites the pixel stored with a value, even though discard is used.
Last edit: 6 years 1 month ago by OtisInf.

Please Log in or Create an account to join the conversation.

More
6 years 1 month ago - 6 years 1 month ago #2 by crosire
Replied by crosire on topic Bugs in generated HLSL code (2.0.3f1)

gives on the tex2Dfetch line:
error X3017: '__tex2Dfetch': cannot implicitly convert from 'const int2' to 'int4'

That's a known issue and fixed already. I had a parameter mismatch of tex2Dfetch between the parser and HLSL code generator. Should be "tex2Dfetch(sampler2D, int4)" (where .w is the mipmap level), but it's "tex2Dfetch(sampler2D, int2)" in 2.0.3.

So I want to NOT write to the render target if the key isn't ON, so discard the pixel. However discard here doesn't discard the pixel, it will write black (0,0,0,0) to the texture as if it's re-initialized each time.

That's because each rendertarget is cleared to zero before it is bound. This was a requirement for SMAA to work.

not sure, but it looks like 'clip' isn't a known function? I'm using Dx11, on a 780, which IMHO is way inside the required shader model.

(edit) I see 'clip' isn't in the list of known intrinsic functions for reshade, is this because there's no equivalent under OpenGL? I can live without using clip, I just want to discard the complete rendering of the pixel so it will keep the previous value written to the texture. I recon this isn't possible?

ReShade FX has no "clip", because the exact same behavior can be achieved with a conditional discard, which is much clearer (it's obvious the shader code won't continue after a discard statement, not so much after a function call). Keep in mind, you don't write HLSL, even though it looks similar.
Last edit: 6 years 1 month ago by crosire.

Please Log in or Create an account to join the conversation.

More
6 years 1 month ago #3 by OtisInf
Replied by OtisInf on topic Bugs in generated HLSL code (2.0.3f1)

gives on the tex2Dfetch line:
error X3017: '__tex2Dfetch': cannot implicitly convert from 'const int2' to 'int4'

That's a known issue and fixed already. I had a parameter mismatch of tex2Dfetch between the parser and HLSL code generator. Should be "tex2Dfetch(sampler2D, int4)" (where .w is the mipmap level), but it's "tex2Dfetch(sampler2D, int2)" in 2.0.3.

Ok thanks, will work around this.

So I want to NOT write to the render target if the key isn't ON, so discard the pixel. However discard here doesn't discard the pixel, it will write black (0,0,0,0) to the texture as if it's re-initialized each time.

That's because each rendertarget is cleared to zero before it is bound. This was a requirement for SMAA to work.

I feared as much. However it's then a blur to me how I can preserve a value across frames. In another thread you gave the advice to use a 1x1 texture for this. However if that's cleared before it's bound, I don't know what else to use as the clearing before bound makes it lose the values I wanted to preserve across frames ;) I obviously miss something like a flag to set somewhere to make it not being cleared or something else. Any advice would be great :)

not sure, but it looks like 'clip' isn't a known function? I'm using Dx11, on a 780, which IMHO is way inside the required shader model.

(edit) I see 'clip' isn't in the list of known intrinsic functions for reshade, is this because there's no equivalent under OpenGL? I can live without using clip, I just want to discard the complete rendering of the pixel so it will keep the previous value written to the texture. I recon this isn't possible?

ReShade FX has no "clip", because the exact same behavior can be achieved with a conditional discard, which is much clearer (it's obvious the shader code won't continue after a discard statement, not so much after a function call). Keep in mind, you don't write HLSL, even though it looks similar.

Ah thanks for the explanation. Now if I can get the value to be preserved it would be great...

Please Log in or Create an account to join the conversation.

More
6 years 1 month ago - 6 years 1 month ago #4 by crosire
Replied by crosire on topic Bugs in generated HLSL code (2.0.3f1)

I feared as much. However it's then a blur to me how I can preserve a value across frames. In another thread you gave the advice to use a 1x1 texture for this. However if that's cleared before it's bound, I don't know what else to use as the clearing before bound makes it lose the values I wanted to preserve across frames ;) I obviously miss something like a flag to set somewhere to make it not being cleared or something else. Any advice would be great :)

The ugly workaround solution would be to create a second "backup" texture. So:
  • pass 1 [RenderTarget = texture 2]: copy texture 1 to texture 2
  • pass 2 [RenderTarget = texture 1]: instead of discard, read the data you want to preserve from texture 2
Last edit: 6 years 1 month ago by crosire.
The following user(s) said Thank You: OtisInf

Please Log in or Create an account to join the conversation.

More
6 years 1 month ago #5 by OtisInf
Replied by OtisInf on topic Bugs in generated HLSL code (2.0.3f1)

I feared as much. However it's then a blur to me how I can preserve a value across frames. In another thread you gave the advice to use a 1x1 texture for this. However if that's cleared before it's bound, I don't know what else to use as the clearing before bound makes it lose the values I wanted to preserve across frames ;) I obviously miss something like a flag to set somewhere to make it not being cleared or something else. Any advice would be great :)

The ugly workaround solution would be to create a second "backup" texture. So:
  • pass 1 [RenderTarget = texture 2]: copy texture 1 to texture 2
  • pass 2 [RenderTarget = texture 1]: instead of discard, read the data you want to preserve from texture 2


Heh thanks :) Will try it out.

Please Log in or Create an account to join the conversation.

More
6 years 1 month ago #6 by OtisInf
Replied by OtisInf on topic [SOLVED] Bugs in generated HLSL code (2.0.3f1)
Works like a charm! :)

Ok, so to recap this thread: problem 1 is already solved in a new build, problem 2 has a workaround, problem 3 (clip) is not supported. Solved :)

Please Log in or Create an account to join the conversation.

  1. Forum
  3. ReShade
  5. Troubleshooting
  7. [SOLVED] Bugs in generated HLSL code (2.0.3f1)