So, for the game Warhammer 40k: Dawn of War - Dark Crusade I want to modify the SMAA shader to use the depthbuffer as a mask because I want to take advantage of the engine quirk that all the text (and only the text) is completely black in the depth buffer.
So, if I could use it as a mask (where the changes to the pixel are discarded if the depthbuffer at that pixel is compeltely black), SMAA won't blur the text very much (if at all).
I've already accomplished the above for qUINT's MXAO shader and it greatly improved text readability in this game. This was VERY straightforward due to how it's coded (it was literally one added line of code + the #define to make it a reshade menu toggle).
void PS_StencilSetup(in MXAO_VSOUT MXAO, out float4 color : SV_Target0)
{
float distance = qUINT::linear_depth(MXAO.uv.zw);
if( distance >= MXAO_FADE_DEPTH_END
#if(MXAO_MASK_CLOSE != 0)
|| distance == 0.0
#endif
|| 0.25 * 0.5 * MXAO_SAMPLE_RADIUS / (tex2D(sMXAO_DepthTex, MXAO.uv.zw).x + 2.0) * BUFFER_HEIGHT < 1.0
|| MXAO.uv.z > 1.0
|| MXAO.uv.w > 1.0
) discard;
color = 1.0;
}
It's been years since I coded shaders and SMAA.fx and SMAA.fxh are a bit confusing to me due to all the #define aliases and I'm rusty.
Basically all I want to do is sample the depth texture, test if it's perfectly black, and then discard if it is. I can probably do better than using a discard, but whatever.
I'm thinking this is the function within SMAA.fx that I want to modify, using depthLinearSampler:
float2 SMAAEdgeDetectionWrapPS(
float4 position : SV_Position,
float2 texcoord : TEXCOORD0,
float4 offset[3] : TEXCOORD1) : SV_Target
{
// do stuff here: sample depthLinearSampler, test if the pixel perfectly black, and then discard if it is
if (EdgeDetectionType == 0 && SMAA_PREDICATION == true)
return SMAALumaEdgePredicationDetectionPS(texcoord, offset, colorGammaSampler, depthLinearSampler);
else if (EdgeDetectionType == 0)
return SMAALumaEdgeDetectionPS(texcoord, offset, colorGammaSampler);
if (EdgeDetectionType == 2)
return SMAADepthEdgeDetectionPS(texcoord, offset, depthLinearSampler);
if (SMAA_PREDICATION)
return SMAAColorEdgePredicationDetectionPS(texcoord, offset, colorGammaSampler, depthLinearSampler);
else
return SMAAColorEdgeDetectionPS(texcoord, offset, colorGammaSampler);
}
Can anyone throw me a bone?
Thanks in advance.
Topic Author