3D Depth Map Based Stereoscopic Shader
- BlueSkyKnight
-
Topic Author
Aelius Maximus wrote:
The current WIP shader there is already a decent 3d image, it's just that the distance between near and far objects does not quite look right yet, but as i said previously this first rendition is very promising and i think it holds a lot of potential, not to mention any app/game ReShade supports will be capable of 3d conversion with no need for depth map, also it will be simple to configure, or will require no configuration at all.
.
The depth map I'm trying to make has a lot of issues. Because of Light sources . Potential yes but as of right now it's out of my league. the research done by other people are done in teams. For this kind of thing I would need help.
Aelius Maximus wrote:
I can't even comprehend how complex the process of converting something like the methods you mention into HLSL would be. Is there any way to obtain the code from LG's 2DTO3DCONVERSION? Or something similar to it? Finding that would be a goldmine i guess.
.
This would be nice to get my hands on this code since. It would help out a lot to see how some one did it before.
Aelius Maximus wrote: So from what i gather the spear back to plane process, seems like the most likely scenario for the 2d to 3d process for this shader in the future?
.
So as you see with the shader I have out now uses a inverted image as a assistant to the Presudo depth map. To many flaws in this approach.
The other way I tried was RGBtoHSV then Gray Scale the image. Then I ran a Edge Detection on the back buffer then filled in the space between the edges. Then combined the two and ran a a 10x Median filter. To give a odd looking depth Depth map. This was better then the one I posted. But, it's very intensive as a shader.
Determining what is "in front" and what is "in the background." Is a problem. That I can't solve right now.
If I can't figure out something I try to forget it for sometime then comeback too it with a new mind. Since a problem like this is very complicated.
Aelius Maximus wrote: As for the pinhole cam model, and mapping the 3d to 2d coordinates the other way around, is the equation behind the paywall? Or at least some research that will help you to work out the reverse equation of mapping 3d to 2d coordinates? Maybe i can help with the paywall if you still need to get past it in the form of a donation i think. But i will need some help at some point with my current attempts at back-porting the WIP 2dto3d shader, To really use it's potential, i want to try it with a few 2d apps/media players that reshade 3.0 can't be used with because of the overlay(things like vlc, MPC, Google Earth/Streetview, the overlay cannot be accessed to even enable the shader)
.
I don't know what kind of work or if just a idea they had. I don't know it if will be any use to to me. So it's not worth the price Since I can't even glance at it.
Aelius Maximus wrote: Edit: I have already managed to get the shader to compile with ReShade 2.0 but still too many "implicit truncation of vector type" errors that i don't know how to correct. If you can help i will send a more substantial donation your way due to your increasingly hard work and experimentation with these shaders. As i know backporting is additional work on your part.
.
The code a mess Because It's just a mess of ideas done quickly. I have not even uploaded my 2nd attempt. Too many errors to count. But, man if we had access to how LG 2d to 3d Conversion works.... This would be a lot more help.
Please Log in or Create an account to join the conversation.
- BlueSkyKnight
-
Topic Author
Did you get to try the checkerboard 3D shader to see if it works on your TV? I would like to know. It's in the Alternative Shader folder with a readMe.
It's called SuperDepth3D_Reprojection_3DC.fx
Please Log in or Create an account to join the conversation.
- helifax
-
BlueSkyKnight wrote: Another idea is too use the pin hole cam model. There is already a equation for maping 3D to 2D coordinates. I want to do it the other way around. en.wikipedia.org/wiki/Pinhole_camera_mod...f_the_pinhole_camera
This not easy to do for me because I need to learn basic of trig. So Im looking ta basics trigonometry youtube videos. To help me out.
Also looks like some people already did this research .
ieeexplore.ieee.org/document/6288160/
But, it's behind a pay wall.............................................
I was wondering, if I can make you a donation to buy that PDF;)
I could also buy it and send it your way;) (I think my IEEE membership expired, but I would still be happy to send/buy it if you want it to

After all, is for Science and Future!


Best Regards,
Helifax
Please Log in or Create an account to join the conversation.
- Aelius Maximus
-
Please Log in or Create an account to join the conversation.
- BlueSkyKnight
-
Topic Author
helifax wrote: I was wondering, if I can make you a donation to buy that PDF;)
I could also buy it and send it your way;) (I think my IEEE membership expired, but I would still be happy to send/buy it if you want it to)
After all, is for Science and Future!I am just asking how you want to do it
Best Regards,
Helifax
My worry is that this method may not be possible with just a shader.
2D-to-3D conversion for single-view image based on camera projection model and dark channel model
Abstract:
This paper presents a novel technique for the depth estimation from the monocular outdoor images. This technique combines three major components: the camera projection model, object classifier, and dark channel modifier. The camera projection model estimates a reasonable relative depth by the camera parameters from the exchangeable image file format (EXIF). The object classifier categorizes objects into four types; sky, ground, man-made and natural objects, and then assigns initial depth to each object. Finally, the initial depth is further corrected by dark channel model. The experiments show our estimated depth map is close to the ground truth, as well as providing the satisfying stereo visual results.
The things that we will have problems with would be the "he object classifier categorizes objects into four types; sky, ground, man-made and natural objects, and then assigns initial depth to each object."
How would I program some thing like this in the shader. It will fall apart if the Sky is red or any other color then blue and black.
Let me link you guys to what I was reading before. www.slideshare.net/editorijritcc1/depth-...-using-scene-feature
I did try this. The Result was ok... But, seems like they left out the last part.
Please Log in or Create an account to join the conversation.
- BlueSkyKnight
-
Topic Author
We can do every thing in this image
RGB to HSV conversion and Gray-scale conversion.
float3 RGBtoHSV(float2 RGB)
{
float3 HCV = tex2D(BackBuffer,RGB).rgb;
float S = HCV.y / (HCV.z + 0.5);
float3 gray_scale = float3(HCV.x, S, HCV.z);
return dot(gray_scale, float3(0.3, 0.59, 0.11));//Gray-scale conversion.
}
Canny Edge Detection
float4 ED(float2 texcoord) //canny edge detection lifted from nivida
{
float4 thresh=0.1;//change this to fill in the spaces between the edges. Not perfect.....
float3 magdir = tex2D(BackBuffer,texcoord);
float alpha = (0.5/sin(3.14159/8)); // eight directions on grid has issues in rehade when maping it to .x and .y
float4 fwdneighbour, backneighbour;
fwdneighbour = tex2D(BackBuffer,texcoord + 0.5 * pix);
backneighbour = tex2D(BackBuffer,texcoord - 0.5 * pix);
float4 color;
if ( fwdneighbour.z > magdir.z || backneighbour.z > magdir.z )
color = float4(0.0, 0.0, 0.0, 0.0); // not an edgel
else
color = float4(1.0, 1.0, 1.0, 1.0); // is an edgel
if ( magdir.z < thresh.x )
color = float4(0.0, 0.0, 0.0, 0.0); // thresholding
return 1 - color;
}
5 times Median filter to fill in holes. Should be 10 times but slows down the shader way too much.
#define s2(a, b) temp = a; a = min(a, b); b = max(temp, b);
#define mn3(a, b, c) s2(a, b); s2(a, c);
#define mx3(a, b, c) s2(b, c); s2(a, c);
#define mnmx3(a, b, c) mx3(a, b, c); s2(a, b); // 3 exchanges
#define mnmx4(a, b, c, d) s2(a, b); s2(c, d); s2(a, c); s2(b, d); // 4 exchanges
#define mnmx5(a, b, c, d, e) s2(a, b); s2(c, d); mn3(a, c, e); mx3(b, d, e); // 6 exchanges
#define mnmx6(a, b, c, d, e, f) s2(a, d); s2(b, e); s2(c, f); mn3(a, b, c); mx3(d, e, f); // 7 exchanges
float4 Median(float2 texcoord : TEXCOORD)
{
float4 color;
float v[6];
v[0] = comb(texcoord.xy + float2(-1.0, -1.0) * 2.5 * pix);
v[1] = comb(texcoord.xy + float2( 0.0, -1.0) * 2.5 * pix);
v[2] = comb(texcoord.xy + float2(+1.0, -1.0) * 2.5 * pix);
v[3] = comb(texcoord.xy + float2(-1.0, 0.0) * 2.5 * pix);
v[4] = comb(texcoord.xy + float2( 0.0, 0.0) * 2.5 * pix);
v[5] = comb(texcoord.xy + float2(+1.0, 0.0) * 2.5 * pix);
float temp;
mnmx6(v[0], v[1], v[2], v[3], v[4], v[5]);
v[5] = comb(texcoord.xy + float2(-1.0, +1.0) * 2.5 * pix);
mnmx5(v[1], v[2], v[3], v[4], v[5]);
v[5] = comb(texcoord.xy + float2( 0.0, +1.0) * 2.5 * pix);
mnmx4(v[2], v[3], v[4], v[5]);
v[5] = comb(texcoord.xy + float2(+1.0, +1.0) * 2.5 * pix);
mnmx3(v[3], v[4], v[5]);
color = v[4];
return color;
}
Here is the problem What the equation they used for Depth Estimation???
*Side Note*
after reading it it seem like they use a external program that does the depth estimation. Called Make3D
make3d.cs.cornell.edu/data.html
www.cs.cornell.edu/~asaxena/learningdepth/ <-- seem to be research link. In there.......
It steams like www.cs.cornell.edu/~asaxena/learningdept...PS_LearningDepth.pdf Has the answer.... I am going to save this PDF and study it.
Please Log in or Create an account to join the conversation.
- helifax
-
"Make3D v1.0 (Linux and Windows)
(Coming soon.)"
Not very promising:( I wonder how maintained that site is seeing everything is from 2007.... (almost 10 years ago now:( )
EDIT:
I remember reading the PDF a while ago and that Laplacian function didn't work for me as described in the PDF:( But, maybe I missed something?
Please Log in or Create an account to join the conversation.
- BlueSkyKnight
-
Topic Author
helifax wrote: Code section:
"Make3D v1.0 (Linux and Windows)
(Coming soon.)"
Not very promising:( I wonder how maintained that site is seeing everything is from 2007.... (almost 10 years ago now:( )
EDIT:
I remember reading the PDF a while ago and that Laplacian function didn't work for me as described in the PDF:( But, maybe I missed something?
It seems from what I'm reading looks like Laplacian model needs to be assisted. "6The absolute depth features xir are non-negative; thus, the estimated σ
2
1r is also non-negative." This may be where a texture that has a color gradient from black to white may help. Since this part mainly estimated.
I know this because some 2D to 3D converters cheat by using a texture over lay that automatically picked out depending on the the dominant shape in the image.
This is part of the reason i used this texture.
This texture was based off the shapes in this PDF that why it has a round and cross pattern.
cvcl.mit.edu/Papers/Torralba-Oliva02.pdf
Also looks like the people at SHARP looked at this research
www.yumpu.com/en/document/view/18050951/...ent-for-viewer-scien
Looks like HSV color conversion is essential for disparity range.
*Side Note*
Do you still have the code for the Laplacian function? This will save me time in trying to program it my self.
Please Log in or Create an account to join the conversation.
- Aelius Maximus
-
*Side Note*
Do you still have the code for the Laplacian function? This will save me time in trying to program it my self.
A lot of this is way over my head, but just trying to help, would this be of any use in any way?
github.com/vva4innovation/Laplacian-Operator
It's in C++
Please Log in or Create an account to join the conversation.
- BlueSkyKnight
-
Topic Author
Aelius Maximus wrote:
*Side Note*
Do you still have the code for the Laplacian function? This will save me time in trying to program it my self.
A lot of this is way over my head, but just trying to help, would this be of any use in any way?
github.com/vva4innovation/Laplacian-Operator
It's in C++
also uses the OpenMesh Library.... there are some things in there that have to be looked up.
Please Log in or Create an account to join the conversation.
- Aelius Maximus
-
BlueSkyKnight wrote:
Aelius Maximus wrote:
*Side Note*
Do you still have the code for the Laplacian function? This will save me time in trying to program it my self.
A lot of this is way over my head, but just trying to help, would this be of any use in any way?
github.com/vva4innovation/Laplacian-Operator
It's in C++
also uses the OpenMesh Library.... there are some things in there that have to be looked up.
I also found this in a quick lookup, they are in HLSL, Laplace is in there further down the page
www.garagegames.com/community/resources/view/15648
Please Log in or Create an account to join the conversation.
- BlueSkyKnight
-
Topic Author
Aelius Maximus wrote:
BlueSkyKnight wrote:
Aelius Maximus wrote:
*Side Note*
Do you still have the code for the Laplacian function? This will save me time in trying to program it my self.
A lot of this is way over my head, but just trying to help, would this be of any use in any way?
github.com/vva4innovation/Laplacian-Operator
It's in C++
also uses the OpenMesh Library.... there are some things in there that have to be looked up.
I also found this in a quick lookup, they are in HLSL, Laplace is in there further down the page
www.garagegames.com/community/resources/view/15648
I will try this out tomorrow. I need to work on my own shader since I found out there is no reason for me to render the Depth map I have at the full resolution. I can get away with /2 the res.

Please Log in or Create an account to join the conversation.
- Aelius Maximus
-
BlueSkyKnight wrote:
Aelius Maximus wrote:
BlueSkyKnight wrote:
Aelius Maximus wrote:
*Side Note*
Do you still have the code for the Laplacian function? This will save me time in trying to program it my self.
A lot of this is way over my head, but just trying to help, would this be of any use in any way?
github.com/vva4innovation/Laplacian-Operator
It's in C++
also uses the OpenMesh Library.... there are some things in there that have to be looked up.
I also found this in a quick lookup, they are in HLSL, Laplace is in there further down the page
www.garagegames.com/community/resources/view/15648
I will try this out tomorrow. I need to work on my own shader since I found out there is no reason for me to render the Depth map I have at the full resolution. I can get away with /2 the res.
looking at this research also has the side affect to helping the shader we have now.
1/2 the resolution would give you double the performance increase right?
It's good to see that the 2dto3d might be moving along again, thought it was a non starter for a second there

Hopefully those two small blocks of code for Laplace are of some help.
On another note, i've gotten my terrible backporting job of your current WIP shader down to 4 (implicit truncation of vector type) yellow warning errors
it is these 4 lines and it is column 10 for each one, so "DepthL" and "DepthR" are the particular definitions that ReShade 2.0 is complaining about, wether it has something to do with the rest of the string of code after column 10 i am not sure. I copied and pasted each line individually, i guess it will be the same solution for each line as they are more or less the same, so hopefully will be easy for you

DepthL = min(DepthL,(tex2D(PseudoDofSamplerS,float2(texcoord.x+uv.x*pix.x, texcoord.y)).r)-tex2D(SamplerCC,float2(texcoord.x, texcoord.y)).rgb);
DepthR = min(DepthR,(tex2D(PseudoDofSamplerS,float2(texcoord.x-uv.x*pix.x, texcoord.y)).r)-tex2D(SamplerCC,float2(texcoord.x, texcoord.y)).rgb);
DepthL = min(DepthL,1 - (tex2D(PseudoDofSamplerC,float2(texcoord.x+uv.x*pix.x, texcoord.y)).r)-tex2D(SamplerCC,float2(texcoord.x, texcoord.y)).rgb);
DepthR = min(DepthR,1 - (tex2D(PseudoDofSamplerC,float2(texcoord.x-uv.x*pix.x, texcoord.y)).r)-tex2D(SamplerCC,float2(texcoord.x, texcoord.y)).rgb);
I quite literally have no idea what to change, sorry to bother you again about this.
Please Log in or Create an account to join the conversation.
- BlueSkyKnight
-
Topic Author
Removed a unnecessary pass and removed the 2nd Depth map view.
Now I am just running the depth map view from the main Depth Map. No FPS improvements. But, it's cleaner code.
Now I also halved the resolution on the depth map. FPS improvement here less memory usage. Learned this from research done on depth map Reconstruction.
Also changed the code for the 3D Checkerboard Pattern. I'm sure it works now. you may have to set your resolution to native for this to work.
Other then that no change to the depth maps yet.
Up coming updates for 1.8.4.
Working on the weapon issue in some games. Like Warhamer Vermintide.
Here the code snip that will be added.
float Adj;
if (WeaponDM == 0)
{
Adj = 0;
WDM = 0;
}
if (WeaponDM == 1)
{
Adj = 1.5;
float cF = 1.5;
float cN = 1.5;
WDM = log(cF * cN/WDM - cF);
}
if (WeaponDM == 2)
{
Adj = 1.5;
float cF = 1.5;
float cN = 1.5;
WDM = 1 - (log(cF * cN/WDM - cF));
}
if (WeaponDM == 3)
{
Adj = Near;
float cF = 1.5;
float cN = 1.5;
WDM = log(cF * cN/WDM - cF);
}
if (WeaponDM == 4)
{
Adj = Near;
float cF = 1.5;
float cN = 1.5;
WDM = 1 - log(cF * cN/WDM - cF);
}
float4 D = lerp(depthM,WA,WDM*Adj);
This is still work in progress and this may change a bit. As you can see I am useing lerp to performs a linear interpolation between the two depth maps. I don't know the performance impact. But, It will be tested when I have time. At best no impact. At worst maybe we lose the FPS boost gained by rendering the depth map at half rez.
Please Log in or Create an account to join the conversation.
- BlueSkyKnight
-
Topic Author
Aelius Maximus wrote:
BlueSkyKnight wrote:
Aelius Maximus wrote:
BlueSkyKnight wrote:
Aelius Maximus wrote:
*Side Note*
Do you still have the code for the Laplacian function? This will save me time in trying to program it my self.
A lot of this is way over my head, but just trying to help, would this be of any use in any way?
github.com/vva4innovation/Laplacian-Operator
It's in C++
also uses the OpenMesh Library.... there are some things in there that have to be looked up.
I also found this in a quick lookup, they are in HLSL, Laplace is in there further down the page
www.garagegames.com/community/resources/view/15648
I will try this out tomorrow. I need to work on my own shader since I found out there is no reason for me to render the Depth map I have at the full resolution. I can get away with /2 the res.
looking at this research also has the side affect to helping the shader we have now.
1/2 the resolution would give you double the performance increase right?
It's good to see that it might be moving along again, thought it was a non starter for a second there
Hopefully those two small blocks of code for Laplace are of some help.
On another note, i've gotten my terrible backporting job of your current WIP shader down to 4 (implicit truncation of vector type) yellow warning errors
it is these 4 lines and it is column 10 for each one, so "DepthL" and "DepthR" are the particular definitions that ReShade 2.0 is complaining about, wether it has something to do with the rest of the string of code after column 10 i am not sure. I copied and pasted each line individually, i guess it will be the same solution for each line as they are more or less the same, so hopefully will be easy for you
DepthL = min(DepthL,(tex2D(PseudoDofSamplerS,float2(texcoord.x+uv.x*pix.x, texcoord.y)).r)-tex2D(SamplerCC,float2(texcoord.x, texcoord.y)).rgb);
DepthR = min(DepthR,(tex2D(PseudoDofSamplerS,float2(texcoord.x-uv.x*pix.x, texcoord.y)).r)-tex2D(SamplerCC,float2(texcoord.x, texcoord.y)).rgb);
DepthL = min(DepthL,1 - (tex2D(PseudoDofSamplerC,float2(texcoord.x+uv.x*pix.x, texcoord.y)).r)-tex2D(SamplerCC,float2(texcoord.x, texcoord.y)).rgb);
DepthR = min(DepthR,1 - (tex2D(PseudoDofSamplerC,float2(texcoord.x-uv.x*pix.x, texcoord.y)).r)-tex2D(SamplerCC,float2(texcoord.x, texcoord.y)).rgb);
I quite literally have no idea what to change, sorry to bother you again about this.
I really should have left notes. As you can see it was a rushed job.
I'm sorry about that.
maybe if you change rgb to just r since the rest of the colors are not necessary.
Also most of the problems come not using passes when passing information from one to the other.
convert them to void and create a pass to a texture. Then Pass that texture over. This way you may get a performance boost. maybe not. also reshade will stop yelling at you..
you can also remove
DepthL = min(DepthL,1 - (tex2D(PseudoDofSamplerC,float2(texcoord.x+uv.x*pix.x, texcoord.y)).r)-tex2D(SamplerCC,float2(texcoord.x, texcoord.y)).rgb);
DepthR = min(DepthR,1 - (tex2D(PseudoDofSamplerC,float2(texcoord.x-uv.x*pix.x, texcoord.y)).r)-tex2D(SamplerCC,float2(texcoord.x, texcoord.y)).rgb);
PseudoDofSamplerC is not that good.
Please Log in or Create an account to join the conversation.
- zig11727
-
Yes How do I install it ?
When I copy it to the game directory where exe is located it doesn't show up in the Reshade menu.
I got it the show in the menu and when I select shader for checkerboard I get a black screen.
Please Log in or Create an account to join the conversation.
- crosire
-
Download latest ReShade 3.0 Beta from reshade.me/#download (under "Alternative Downloads"). Install it using the setup tool. Download SuperDepth3D_Reprojection.fx from GitHub and put that file into the same directory the ReShade DLL was installed to (not the Shaders folder, the fx file!). Done.zig11727 wrote: Yes How do I install it ?
Alternativly you can put SuperDepth3D_Reprojection.fx somewhere else and simply add that directory to the effect search path list on the 'Settings' tab of the ReShaed in-game menu.
Please Log in or Create an account to join the conversation.
- BlueSkyKnight
-
Topic Author
zig11727 wrote: @BlueSkyKnight
Yes How do I install it ?
When I copy it to the game directory where exe is located it doesn't show up in the Reshade menu.
I got it the show in the menu and when selected everything goes to a black screen.
Ya I tested it in DX9 Game seems like some strange code problem. I am updating the test Shader with the work around.
Checkerboard 3D Pixel Size should be set to 1 increase it.
Please Log in or Create an account to join the conversation.
- otherman
-
what is static is Hud, what moves slow is far, what moves fast is near. This combined with what you already have, may bring pretty neat results.
Please Log in or Create an account to join the conversation.
- mr_spongeworthy
-
Please Log in or Create an account to join the conversation.