DisplayDepth normal map generator

  • Posts: 156
1 year 14 hours ago - 11 months 3 weeks ago #1 by Fu-Bama
There is something off with DisplayDepth normal map generator and don't know what :dry: .

Based of that normal map I created Matcap shader, but then noticed sampling is off, like if the normal map was rotated slightly in X and Y axis.


Source Matcap and result in ReShade




Here's the Matcap shader:
github.com/Fubaxiusz/fubax-shaders/blob/...er/Shaders/Matcap.fx
(the shader is still missing blending options and displays wrong flat surfaces)


*Update
Grab Reflection shader instead:
github.com/Fubaxiusz/fubax-shaders/blob/...haders/Reflection.fx
and Matcap texture:
github.com/Fubaxiusz/fubax-shaders/blob/.../Textures/matcap.png

Please Log in or Create an account to join the conversation.

  • Posts: 3638
11 months 4 weeks ago - 11 months 4 weeks ago #2 by crosire
Keep in mind that those normals are approximations and in screen space (so they are not independent from the camera position). They are not anywhere close to actual vertex normals in world space, so are really useful mostly only to gather spatial information.
To get world space normals you would need to know the current view matrix of the game. ReShade has no way to get that information.

Please Log in or Create an account to join the conversation.

  • Posts: 156
11 months 4 weeks ago - 11 months 4 weeks ago #3 by Fu-Bama
Yes, they are in screen space and that kind of normals are needed for MatCap, but for some reason, even doe depth on the sphere is uniform, normals directing -X (left) are always weaker.
Thought it's because the way depth is sampled in normal map generator, where it samples depth at current pixel position and one on the right, calculating surface angle. But that's not the case apparently.

Please Log in or Create an account to join the conversation.

  • Posts: 3638
11 months 4 weeks ago #4 by crosire
Might be related to depth precision. Maybe also try to increase the offset used to calculate the the vertex position deltas.

Please Log in or Create an account to join the conversation.

  • Posts: 156
11 months 4 weeks ago - 11 months 3 weeks ago #5 by Fu-Bama
I found the fix for bug in normal pass generator.
Apparently coordinates where not centered, here's the fixed algorithm:
float3 NormalVector(float2 texcoord)
{
	float3 offset = float3(ReShade::PixelSize.xy, 0.0);
	float2 posCenter = texcoord.xy;
	float2 posNorth  = posCenter - offset.zy;
	float2 posEast   = posCenter + offset.xz;

	float3 vertCenter = float3(posCenter - 0.5, 1) * GetDepth(posCenter);
	float3 vertNorth  = float3(posNorth - 0.5,  1) * GetDepth(posNorth);
	float3 vertEast   = float3(posEast - 0.5,   1) * GetDepth(posEast);

	return normalize(cross(vertCenter - vertNorth, vertCenter - vertEast)) * 0.5 + 0.5;
}
...and created pull request for the normal generator in DisplayDepth.

For proof fake ball-depth:
Warning: Spoiler! [ Click to expand ]

Then mapped texture accordingly to the normal vector from DisplayDepth (original):
Warning: Spoiler! [ Click to expand ]

And now with fixed normal map:
Warning: Spoiler! [ Click to expand ]

Please Log in or Create an account to join the conversation.

  • Posts: 176
11 months 3 weeks ago #6 by Daodan
Have you tested the normal vector calculation from MXAO (in PS_InputBufferSetup()) with your MatCap shader? It takes all four neighbouring pixels into account and not only two like DisplayDepth.

Please Log in or Create an account to join the conversation.

  • Posts: 156
11 months 3 weeks ago - 11 months 3 weeks ago #7 by Fu-Bama

Daodan wrote: Have you tested the normal vector calculation from MXAO (in PS_InputBufferSetup()) with your MatCap shader? It takes all four neighbouring pixels into account and not only two like DisplayDepth.

It did not work for me. But I already found the bug fix, see post above

Please Log in or Create an account to join the conversation.

  • Posts: 1225
11 months 3 weeks ago - 11 months 3 weeks ago #8 by Marty McFly

Daodan wrote: Have you tested the normal vector calculation from MXAO (in PS_InputBufferSetup()) with your MatCap shader? It takes all four neighbouring pixels into account and not only two like DisplayDepth.


MXAO already has centered coordinates like in Fu-Bama's fix (although coord * 2 - 1 instead of coord - 0.5 so it's the equivalent of 90 degree FoV instead of 45). This bug must've escaped my attention when I updated the DisplayDepth shader... sorry for that. To my understanding, the DisplayDepth is only supposed for debugging purposes and as long as the depth buffer is correct, the normals display something. To be 100% correct, one would need to flip Y axis as well (that's why MXAO output is differently colored than DisplayDepth). MXAO also has a FoV estimation thingy in the vertex shader that allows for more correct normals when the FoV is known. Maybe overkill for DisplayDepth though.

What you are addressing, I'm checking both sides of the current coord is because that way, you get rid of the discontinuities at object edges, that "embossed" look of the normals is gone then. With an effect like this it might not be so severe but AO at those discontinuities will be black and blur filter doesn't work either.

Please Log in or Create an account to join the conversation.