First of all,
pass LinearDepthPass
{
#if (EdgeDetectionType == 3)
VertexShader = Reshade::PostProcessVS;
PixelShader = LinearDepthPS;
RenderTarget = LinearDepthTex;
ClearRenderTargets = true;
#endif
}the #if .etc are preprocessors, which only initialize once upon compilation and they only 'sees' other preprocessor directives.
Instead of codes that actually runs by execution, they are more like a build script that can help built different code with ease.
so in your implementation, after preprocessor done with the initialization, it will be just
since there isn't a
#define EdgeDetectionType 3
before the check.
And
void LinearDepthPS(
in float4 position : SV_Position,
in float2 texcoord : TEXCOORD0,
out float4 LiniOut)
you're missing the output binding, SV_Target or COLOR .etc. but since reshade wasn't trying to compile the shader from the previous reason, you won't see the error message about it.
And I'm curious about what does it say about stencil buffer from your note?
Also, you can look into hlsl and MS effect framework as reshade syntax share some similarity to it.
