anyway to fix upside down depth buffer

  • Kleio420
  • Topic Author
More
8 years 10 months ago #1 by Kleio420 anyway to fix upside down depth buffer was created by Kleio420
I have a game that the buffer is flipped upside down is there a way to flip it through a shader or is this reshade specific ?

Please Log in or Create an account to join the conversation.

  • crosire
More
8 years 10 months ago #2 by crosire Replied by crosire on topic anyway to fix upside down depth buffer
Yeah, just flip the texture coordinates before sampling from it. Shader thing, not related to ReShade itself =).

Please Log in or Create an account to join the conversation.

  • Kleio420
  • Topic Author
More
8 years 10 months ago #3 by Kleio420 Replied by Kleio420 on topic anyway to fix upside down depth buffer

crosire wrote: Yeah, just flip the texture coordinates before sampling from it. Shader thing, not related to ReShade itself =).

what file is this located in i would like to try and do it myself before asking if anyone could change it for me

Please Log in or Create an account to join the conversation.

  • SpinelessJelly
More
8 years 10 months ago - 8 years 10 months ago #4 by SpinelessJelly Replied by SpinelessJelly on topic anyway to fix upside down depth buffer
I would very much like to know that myself. Also, which are exactly the values to be flipped? I tried fooling around (monkey with a typewriter!) with all texcoord values I found in Util.h in the Common folder, but all I got was black screens. Or half-black screens, which was interesting :)
Last edit: 8 years 10 months ago by SpinelessJelly.

Please Log in or Create an account to join the conversation.

  • crosire
More
8 years 10 months ago - 8 years 10 months ago #5 by crosire Replied by crosire on topic anyway to fix upside down depth buffer
I don't know all the files, it's used multiple times across different source files. To flip you would want to invert the Y texture coordinate:
tex2D(depthtex, float2(texcoord.x, 1.0 - texcoord.y))
Last edit: 8 years 10 months ago by crosire.
The following user(s) said Thank You: Kleio420, SpinelessJelly

Please Log in or Create an account to join the conversation.

  • Kleio420
  • Topic Author
More
8 years 10 months ago - 8 years 10 months ago #6 by Kleio420 Replied by Kleio420 on topic anyway to fix upside down depth buffer
havent been able to find anything like that if anyone who reads this doesnt mind helping pointing me to the right function
Last edit: 8 years 10 months ago by Kleio420.

Please Log in or Create an account to join the conversation.

  • SpinelessJelly
More
8 years 10 months ago - 8 years 10 months ago #7 by SpinelessJelly Replied by SpinelessJelly on topic anyway to fix upside down depth buffer
Soooo.... I think it worked? I tried this on Ziggurat (Unity engine, opengl32.dll, reshade 0.18.4). Fixed Cel shader by making ...\ReShade\CustomFX\Cel.h look like this
...
float3 normals(float2 texcoord) // get normal vector from depthmap
{
	float	deltax = linearlizeDepth(tex2D(RFX_depthColor, float2((texcoord.x + px), 1.0 - texcoord.y)).x) - linearlizeDepth(tex2D(RFX_depthColor, float2((texcoord.x - px), 1.0 - texcoord.y)).x),
		deltay = linearlizeDepth(tex2D(RFX_depthColor, float2(texcoord.x, ((1.0 - texcoord.y) + py))).x) - linearlizeDepth(tex2D(RFX_depthColor, float2(texcoord.x, ((1.0 - texcoord.y) - py))).x);	
...

Also Matso DOF seems to work correctly by changing ...\ReShade\Common\Util.h to
...
// Depth buffer access
	float depth = tex2D(RFX_depthColor, float2(texcoord.x, 1.0 - texcoord.y)).x;
...

Both effects shine through the weapon and HUD but I guess that's the game's problem, not Reshade's. Still haven't managed to flip AO but the indication looks valid, thanks again Crosire.
Last edit: 8 years 10 months ago by SpinelessJelly.
The following user(s) said Thank You: Kleio420

Please Log in or Create an account to join the conversation.

  • Kleio420
  • Topic Author
More
8 years 10 months ago - 8 years 10 months ago #8 by Kleio420 Replied by Kleio420 on topic anyway to fix upside down depth buffer

SpinelessJelly wrote: Soooo.... I think it worked? I tried this on Ziggurat (Unity engine, opengl32.dll, reshade 0.18.4). Fixed Cel shader by making ...\ReShade\CustomFX\Cel.h look like this

...
float3 normals(float2 texcoord) // get normal vector from depthmap
{
	float	deltax = linearlizeDepth(tex2D(RFX_depthColor, float2((texcoord.x + px), 1.0 - texcoord.y)).x) - linearlizeDepth(tex2D(RFX_depthColor, float2((texcoord.x - px), 1.0 - texcoord.y)).x),
		deltay = linearlizeDepth(tex2D(RFX_depthColor, float2(texcoord.x, ((1.0 - texcoord.y) + py))).x) - linearlizeDepth(tex2D(RFX_depthColor, float2(texcoord.x, ((1.0 - texcoord.y) - py))).x);	
...

Also Matso DOF seems to work correctly by changing ...\ReShade\Common\Util.h to
...
// Depth buffer access
	float depth = tex2D(RFX_depthColor, float2(texcoord.x, 1.0 - texcoord.y)).x;
...

Both effects shine through the weapon and HUD but I guess that's the game's problem, not Reshade's. Still haven't managed to flip AO but the indication looks valid, thanks again Crosire.

wow thanks this worked at least for the dof, just being able to see where this is applied is super helpful for getting better at understanding the flow of things ty guys very much

i take it for ssao this is what needs changed
float depth = tex2D(RFX_depthColor, texcoord.xy).x;

from looking at it , looks like the entire ssao.h file needs to be changed to work with this
Last edit: 8 years 10 months ago by Kleio420.

Please Log in or Create an account to join the conversation.

  • SpinelessJelly
More
8 years 10 months ago #9 by SpinelessJelly Replied by SpinelessJelly on topic anyway to fix upside down depth buffer
Yes, I tried changing that line but nothing happened (not true, managed to get some error messages). But maybe I got the syntax wrong, I'm pretty clueless code-wise and logic can only get you that far.

Please Log in or Create an account to join the conversation.

  • Kleio420
  • Topic Author
More
8 years 10 months ago #10 by Kleio420 Replied by Kleio420 on topic anyway to fix upside down depth buffer

SpinelessJelly wrote: Yes, I tried changing that line but nothing happened (not true, managed to get some error messages). But maybe I got the syntax wrong, I'm pretty clueless code-wise and logic can only get you that far.

well from my understanding float depth = (varible, float2(texcoord.xy).x; is how every ssao example ive found on the web uses to get the depth if you look at the rest the same layout is used for specific functions which is why i think it needs either a setting to enable a full on global buffer flip in one line of code or editing the entire file to work with it. Since this seems to be hugely a unity engine issue maybe we can get a fix in a framework release. By no means am i saying im correct in what i just typed as im probably wrong hopefully there is enough interest in this to put a fix for it

Please Log in or Create an account to join the conversation.

  • strelokgunslinger
More
8 years 10 months ago #11 by strelokgunslinger Replied by strelokgunslinger on topic anyway to fix upside down depth buffer
DisplayDepth.h

#if (RFX_DisplayDepth == 1)

NAMESPACE_ENTER(RFX) 

void DisplayDepth(in float4 position : SV_Position, in float2 texcoord : TEXCOORD0, out float3 color : SV_Target)
{
	color.rgb = tex2D(RFX_depthTexColor,texcoord).rgb;
	
}

technique Depth_Tech < bool enabled = false; int toggle = RFX_DepthToggleKey;>
{
	pass
	{
		VertexShader = RFX_VS_PostProcess;
		PixelShader = DisplayDepth;
	}
}


NAMESPACE_LEAVE()

#endif

Where do I input ?

tex2D(depthtex, float2(texcoord.x, 1.0 - texcoord.y))

Please Log in or Create an account to join the conversation.

  • crosire
More
8 years 10 months ago #12 by crosire Replied by crosire on topic anyway to fix upside down depth buffer

strelokgunslinger wrote:

...
color.rgb = tex2D(RFX_depthTexColor,texcoord).rgb;
...

Where do I input ?

tex2D(depthtex, float2(texcoord.x, 1.0 - texcoord.y))

There aren't many lines which look similar =P. Just repalce it there.

Please Log in or Create an account to join the conversation.

  • biliskner
More
8 years 8 months ago - 8 years 8 months ago #13 by biliskner Replied by biliskner on topic anyway to fix upside down depth buffer

crosire wrote:

strelokgunslinger wrote:

...
color.rgb = tex2D(RFX_depthTexColor,texcoord).rgb;
...

Where do I input ?

tex2D(depthtex, float2(texcoord.x, 1.0 - texcoord.y))

There aren't many lines which look similar =P. Just repalce it there.


Thanks crosire, this helped me alot with fixing my Unity depth buffer inversion
I have zero programming knowledge so I spent the better part of the last 24hrs reading up about HLSL just so I could change my reshade.fx file to try and add a more permanent fix.
What i found was that changing the texcoord values manually for each shader effect was very time consuming and difficult to find the right line.. so my fix was to try and add a toggle in common.cfg

I then change RFX_PS_StoreDepth directly
Warning: Spoiler!


after a few more hours of headache where i messed up my graphics -_- I realised DisplayDepth uses RFX_depthTexColor which is saved.. but CEL and SMAA use RFX_depthColor which i THINK? is read directly from the buffer

Long story short.. i added a few more lines of code to affect anything using RFX_depthTexColor or RFX_depthColor. I hope this helps someone, please feel free to correct me if my code is messed up.

Code additions/changes follow
Warning: Spoiler!
Last edit: 8 years 8 months ago by biliskner. Reason: made declaring RFX_fixdepthTex conditional
The following user(s) said Thank You: SpinelessJelly

Please Log in or Create an account to join the conversation.

  • crosire
More
8 years 8 months ago #14 by crosire Replied by crosire on topic anyway to fix upside down depth buffer
Code looks correct, so if you have both the RFX_PS_FixDepth and RFX_PS_StoreDepth changes it should work in all cases.

The current handling is a little complicated, too many different textures and samplers to keep track of. That should be simplified in the next versions.

Please Log in or Create an account to join the conversation.

  • biliskner
More
8 years 8 months ago #15 by biliskner Replied by biliskner on topic anyway to fix upside down depth buffer
Thanks so much for looking through my code crosire!
Yeah learning about textures and samplers was a massive headache.. keeping track of them was another heh

Please Log in or Create an account to join the conversation.

We use cookies
We use cookies on our website. Some of them are essential for the operation of the forum. You can decide for yourself whether you want to allow cookies or not. Please note that if you reject them, you may not be able to use all the functionalities of the site.