3D Depth Map Based Stereoscopic Shader

CEMU and Shifting depth map issue. The Depth Map LOZ:BOTW Seems to shift based on camera position. So even if you find the right values once you move it will shift.

Yes, i noticed this and it's infuriating because otherwise it would work perfectly. It shifts the depth map to a deeper depth when ever the camera points down towards link and at some random angles. I don't suppose you found a solution to this?

The items menu also seems to stop rendering after a few seconds, switching to the save menu and back restores it as well as exiting the menu.

I'm going to use Nsight to do a frame debug. I want to check if this is something related to reshade or happens in engine.

Is it possible to bind textures to other buffers in reshade beside the standard color and depth buffers? I'm pretty sure that BOTW has a separate buffer that it stores depth in for other post effects.

sgsrules wrote: Yes, i noticed this and it's infuriating because otherwise it would work perfectly. It shifts the depth map to a deeper depth when ever the camera points down towards link and at some random angles. I don't suppose you found a solution to this?

No. But, I will try to something later after I get the Depth Values for most the games I listed.

sgsrules wrote: The items menu also seems to stop rendering after a few seconds, switching to the save menu and back restores it as well as exiting the menu.

I'm going to use Nsight to do a frame debug. I want to check if this is something related to reshade or happens in engine.

Is it possible to bind textures to other buffers in reshade beside the standard color and depth buffers? I'm pretty sure that BOTW has a separate buffer that it stores depth in for other post effects.

Here reshade git hub github.com/crosire/reshade if you want to work on this problem.

It looks like BOTW is using dynamic clipping planes to maximize the precision of it's depth buffer. Basically the near and far clipping planes are being moved depending on the camera position as well as the objects that are inside the view frustum. We obviously don't have access to any of this information but we do have access to the depth buffer, so it's possible to figure out when the clip planes are moved since the shifts are generally pretty drastic.

I've managed to come up with a fairly decent solution. Basically i take various samples from the depth buffer and check their depth, if they fall outside certain values then the game has shifted it's clipping planes. I then calculated the difference between these values to calculate a offset for the clipping planes. I then use the new clipping plane values to linearize the depth value using the standard OpenGL depth linearization equation. For the most part it works really well, it's completely removed the sudden changes in the stereoscopic effect that were caused by the camera movement. The only times i've had issues is when objects like grass get between Link and the camera, this causes the depth buffer to shift a bit but's its not that bad. With a bit more work i should be able to fix this as well.

I'll post my results here soon. But before i do that i want to map the depth buffer a bit better, since the adjustments I'm making are dependent on the values used to calculate the linear depth. I'm linearizing the depth buffer using the OpenGL equation i posted with .05 for the near clip plane and 1 for the far clip plane. These are the reference values i use before i alter the clip planes when the depth values shift. These values work ok for the most part, the stereo looks great as long as things are a few feet in from of link. Close than this point things get fairly flat the closer they get. I think i need to adjust the convergense aka zero parralax distance, or maybe adjust the depth mapping. I could play with variables and equations for a few hours but I figured you could point me in the right direction since you've been messing around with this stuff for months.

The OpenGL 3dVision wrapper I'm using only works with reshade 2.0 so I've been basing most of my code on your older 1.89 version. For the most part everything is the same as your newer code except that I noticed you added some AO code. I'm very familiar with AO, I've written a few custom implementations myself but i don't understand how your using it in conjustion with the 3d shader. Care to fill me in? What advantages does it provide?

sgsrules wrote: It looks like BOTW is using dynamic clipping planes to maximize the precision of it's depth buffer. Basically the near and far clipping planes are being moved depending on the camera position as well as the objects that are inside the view frustum. We obviously don't have access to any of this information but we do have access to the depth buffer, so it's possible to figure out when the clip planes are moved since the shifts are generally pretty drastic.

I've managed to come up with a fairly decent solution. Basically i take various samples from the depth buffer and check their depth, if they fall outside certain values then the game has shifted it's clipping planes. I then calculated the difference between these values to calculate a offset for the clipping planes. I then use the new clipping plane values to linearize the depth value using the standard OpenGL depth linearization equation. For the most part it works really well, it's completely removed the sudden changes in the stereoscopic effect that were caused by the camera movement. The only times i've had issues is when objects like grass get between Link and the camera, this causes the depth buffer to shift a bit but's its not that bad. With a bit more work i should be able to fix this as well.

Good to hear. I would love to see it. If it work out for you. Because, It would be a simple way to just have the Shader auto adjust for the correct precision.


Making it easier for people to just turn it on and use it.

sgsrules wrote: I'll post my results here soon. But before i do that i want to map the depth buffer a bit better, since the adjustments I'm making are dependent on the values used to calculate the linear depth. I'm linearizing the depth buffer using the OpenGL equation i posted with .05 for the near clip plane and 1 for the far clip plane. These are the reference values i use before i alter the clip planes when the depth values shift. These values work ok for the most part, the stereo looks great as long as things are a few feet in from of link. Close than this point things get fairly flat the closer they get. I think i need to adjust the convergense aka zero parralax distance, or maybe adjust the depth mapping. I could play with variables and equations for a few hours but I figured you could point me in the right direction since you've been messing around with this stuff for months.

ZPD I this adjustment before but it caused too much warping when I did add it. If you want simple convergence.

ParallaxL= Deviation * (1-ZDP / DepthL);
ParallaxR= Deviation * (1-ZDP / DepthR);

You can do it as shown. But, careful ZPD will cause depth conflicts when close to wall.


Better yet I will add it to the experimental so you can just see it your self.
.

sgsrules wrote: The OpenGL 3dVision wrapper I'm using only works with reshade 2.0 so I've been basing most of my code on your older 1.89 version. For the most part everything is the same as your newer code except that I noticed you added some AO code. I'm very familiar with AO, I've written a few custom implementations myself but i don't understand how your using it in conjustion with the 3d shader. Care to fill me in? What advantages does it provide?

The AO code is to allow for some definition when a model or a object comes really close to the screen. With out it a face would look really flat. Also helps with adding a bit more depth to crevices and a little bit more separation with objects stacked on top of each other.

Now I am writing a new shader "virtual monitor". The shader is intended for headsets for smartphones similar to google cardboard. The user will have to see before his eyes a virtual screen with a high-quality setting. The diagonal of the screen will be adjusted, the actual distance between the lenses in mm, the distance from the lens to the screen of the smartphone, and distortion of the lenses. After that, you can adjust the distance at which you can see the virtual monitor in the headset.
BlueSkyKnight, what kind of headset do you have?

Good to hear. I would love to see it. If it work out for you. Because, It would be a simple way to just have the Shader auto adjust for the correct precision.

It's not something that would work universally, you could possibly use it to avoid certain issues, like a menu popping up at the wrong depth. You need to have a reference object that is generally at the same depth.

ZPD I this adjustment before but it caused too much warping when I did add it. If you want simple convergence.

You need to have the calculation for zero parallax distance for proper 3d, otherwise things get flattened out if as they get closer to the camera. If you're experiencing excessive warping it's because the depth values aren't properly mapped or you're applying to much parallax.

I removed your disocclusion shader function, I'm guessing you put it in there to try to smooth out the halo around objects but it didn't work to well in my tests all it did was add unnecessary warping, especially after i added the zpd adjustment . You can't just blur depth values and use those values on to calulate the stereo offset. If you're trying to smoothen out the halo i would recommend doing the blurring on the final image using the blurred depth - the original depth (basically a sharp mask) as a mask, but this ight lead to other artifacts. I ended up using Crytek's original s3d functions and got better results, no offense. Either way thanks for working on this it has been an invaluable resource.

sgsrules wrote: It's not something that would work universally, you could possibly use it to avoid certain issues, like a menu popping up at the wrong depth. You need to have a reference object that is generally at the same depth.

Still I would love to see what you did.

sgsrules wrote: You need to have the calculation for zero parallax distance for proper 3d, otherwise things get flattened out if as they get closer to the camera. If you're experiencing excessive warping it's because the depth values aren't properly mapped or you're applying to much parallax.

Added that code in to the experimental shader I don't see this problem. It was a warning for you since you would encounter this at first. But, ok.....

sgsrules wrote: I removed your disocclusion shader function, I'm guessing you put it in there to try to smooth out the halo around objects but it didn't work to well in my tests all it did was add unnecessary warping, especially after i added the zpd adjustment . You can't just blur depth values and use those values on to calulate the stereo offset. If you're trying to smoothen out the halo i would recommend doing the blurring on the final image using the blurred depth - the original depth (basically a sharp mask) as a mask, but this ight lead to other artifacts. I ended up using Crytek's original s3d functions and got better results, no offense. Either way thanks for working on this it has been an invaluable resource.

Disocclusion is optional and always will be. Like I said, Lets see what you have done.

This is all good. But, it would be nice to show your work. There is a lot of talk of improvement. I would love to merge this work in to the new shader so that every one here can enjoy the improvements. As it sound you go a lot done.

I want to keep things open for every one here.

ZapaL wrote: Now I am writing a new shader "virtual monitor". The shader is intended for headsets for smartphones similar to google cardboard. The user will have to see before his eyes a virtual screen with a high-quality setting. The diagonal of the screen will be adjusted, the actual distance between the lenses in mm, the distance from the lens to the screen of the smartphone, and distortion of the lenses. After that, you can adjust the distance at which you can see the virtual monitor in the headset.
BlueSkyKnight, what kind of headset do you have?

Good too see people working on these problems.

Also I have the OSVR HMD 1.3 www.osvr.org/hardware/buy/

Thank you BTW.

This is all good. But, it would be nice to show your work. There is a lot of talk of improvement. I would love to merge this work in to the new shader so that every one here can enjoy the improvements. As it sound you go a lot done.

I want to keep things open for every one here.

Oh yes, for sure! I'm all about open source and sharing code because everyone benefits. I'm still experimenting with what works, right now i'm using a lot of "magic numbers" which i hate doing because changing one variable requires changing others. Once I'm happy with the code i'll make sure i post it and share my findings.

Out of curiosity why did you remove the dual-depth-map version? I will have to go back and test to be absolutely certain I'm seeing what I think I'm seeing, but I'm almost certain that I was able to get a bit more depth with a bit less distortion using that version (I'm thinking mostly of Witcher 3 in this case) than the newer version with the AO. It was kind of a pain to try two different depth-maps and then get the blend point correct, but it definitely worked pretty darn well (I'm leaving that version installed in FO4 for now, but may try newer releases when I get time).

mr_spongeworthy wrote: Out of curiosity why did you remove the dual-depth-map version? I will have to go back and test to be absolutely certain I'm seeing what I think I'm seeing, but I'm almost certain that I was able to get a bit more depth with a bit less distortion using that version (I'm thinking mostly of Witcher 3 in this case) than the newer version with the AO. It was kind of a pain to try two different depth-maps and then get the blend point correct, but it definitely worked pretty darn well (I'm leaving that version installed in FO4 for now, but may try newer releases when I get time).

I have been working on 1.9.6 trying to make it easier for people to use and adjust. It just taking me time get all the depth map set for the games. Also, I needed to rewrite the weapon depth map part so I am able to adjust it properly in most games. But, I will go back and look at the setting. Just give me time.

Would it be possible to rearrange line interlaced to top and bottom ?
This would make nvidias 3dvision available to every monitor and hmd, because it only supports line interlaced right now.

And thanks for the great work so far, the shader seems to work well in most games I tested

DerLurch wrote: Would it be possible to rearrange line interlaced to top and bottom ?
This would make nvidias 3dvision available to every monitor and hmd, because it only supports line interlaced right now.

And thanks for the great work so far, the shader seems to work well in most games I tested

I tried to do this already no success since Reshade is picked up before 3D vision converts to stereo.

Using DM4 works pretty well with Elder Scrolls Online but notice the 3D effect occasionally turns off and then back on - think it might be related to the HUD but even when I disable the HUD it still happens. Any idea how to get around this perhaps?

Arioch wrote: Using DM4 works pretty well with Elder Scrolls Online but notice the 3D effect occasionally turns off and then back on - think it might be related to the HUD but even when I disable the HUD it still happens. Any idea how to get around this perhaps?

This is due to Reshade disabling the depth buffer when it detects an online connection or something to that effect.

Wish there was a workaround...game looks really good with this.

It is my understanding that someone here own an OSVR? I was considering getting one so i can view this shader in desktop mirror mode, which is my understanding of how standard SBS content is viewed on this HMD, Are there any negative effects when viewing SBS games through the display using desktop mode? Cropped edges, distortion, stretched image? I attempted to use my Oculus Rift CV1 as an additional display and it has a couple of these problems along with the fact that the SBS content could not be focused on properly (3D image appeared distorted in the background) but not up close. I guess it works to some extent as i remember someone talking about it on here.

Aelius Maximus wrote: It is my understanding that someone here own an OSVR? I was considering getting one so i can view this shader in desktop mirror mode, which is my understanding of how standard SBS content is viewed on this HMD, Are there any negative effects when viewing SBS games through the display using desktop mode? Cropped edges, distortion, stretched image? I attempted to use my Oculus Rift CV1 as an additional display and it has a couple of these problems along with the fact that the SBS content could not be focused on properly (3D image appeared distorted in the background) but not up close. I guess it works to some extent as i remember someone talking about it on here.

Considering the poor quality of my OSVR HMD and the difficulty getting it working I have to say. Do not get the OSVR 1.3 HMD. I have not been able to test out the other HMD from razer. But, as the 1.3 HMD is kind of like the Playstation HMD in image quality. As for cropped edges, distortion and stretched image. Basically, yes you will get this when using SBS. It's not that bad tough. Your Oculus Rift is better than the OSVR HMD. I would stick to it until there is a screen with a better res.

@BlueSkyKnight

Do you own a NVidia 1070 or 1080 GTX if so I would like to donate a key code for a Ubisoft game For Honor or Wild lands.
Also you new shader or the old one doesn't work with Mirror's Edge Catalyst No depth buffer this was one of best looking games with your shader.

zig11727 wrote: @BlueSkyKnight

Do you own a NVidia 1070 or 1080 GTX if so I would like to donate a key code for a Ubisoft game For Honor or Wild lands.
Also you new shader or the old one doesn't work with Mirror's Edge Catalyst No depth buffer this was one of best looking games with your shader.

I own a gtx 670 and a RX 580. I use both to test games.

As for Mirror's Edge Catalyst use 1.9.6 DM 4 and Depth Map Adjustment Range 5.0-10

It's in the experimental folder. I have lowered the number of depth maps. It's just taking me time to test all the games. I am updating it as fast as I can. Also, look at the new Game Settings text. Some of the games listed like, Homefront The Revolution I will need help with because I don't have them.

I will add Help Tag when I get to them.

I will make discord server people can talk to me a bit more directly when it comes to adjusting and making direct requests.