Advice needed for GlideN64 plugin compatibility

Gonetz, who works on the GlideN64 plugin, was looking into an option to make the depth buffer available so that effects like ambient occlusion could work through ReShade. He's run into a problem that requires some clarification about what ReShade needs and uses specifically to work:

I installed latest Reshade and tested it with Project64 2.3
To my surprise, depth based filters do work. I used DisplayDepth as test shader, here how it looks with Release 2.0

No problems, as well as with other depth buffer based shaders.

With current master it looks upside-down:

I tried to invert it, but without success.

The main question: why it ever works? I do not copy depth from FBO to main depth buffer. Release does not do it. Master does not do it too. My only guess was that Reshade takes data not from color and depth buffer but from FBO, because image in FBO is upside down. However, when I apply filters for color buffer such as 'cartoon', it applied correctly, not upside down. So, where Reshade takes depth buffer data?

Basically, the current 2.0 release of the plugin works with MXAO and other effects even though the depth buffer is never deliberately made available, though for some reason MXAO only works when framebuffer emulation is disabled in GlideN64's options. Once FB is turned off, the effect works perfectly, even though depth apparently isn't being copied to the main depth buffer as Gonetz said:


Can anyone clarify what is likely going on here in terms of why MXAO was working, what info it was accessing in order to work, and what Gonetz would specifically need to make available for this and other effects to run? I'd really like for him to be able to get this working, since it's no longer possible to get MXAO working in the newer versions of the plugin and disabling fb emulation in the 2.0 release is a huge downgrade.

ReShade analyzes each framebuffer the app creates and checks if any depth target is bound to it. It does so by looking at any calls to "glFramebufferRenderbuffer", "glFramebufferRenderbufferEXT", "glFramebufferTexture", "glFramebufferTextureARB", "glFrameBufferTextureEXT", "glFramebufferTexture1D", "glFramebufferTexture1DEXT", "glFramebufferTexture2D", "glFramebufferTexture2DEXT", "glFramebufferTexture3D", "glFramebufferTexture3DEXT", "glFramebufferTextureLayer", "glFramebufferTextureLayerARB" and "glFramebufferTextureLayerEXT". If that's the case that framebuffer is added to a list of possible scene depth buffer candidates (relevant code: github.com/crosire/reshade/blob/6064e197...ngl_runtime.cpp#L513 ). Every 30 frames, ReShade goes through that list and searches one with a resolution that is at least 95% of the screen resolution and that was used in the most draw calls per frame (relevant code: github.com/crosire/reshade/blob/6064e197...ngl_runtime.cpp#L878 ). It then rememebers which texture or renderbuffer depth attachment is bound to that framebuffer, so starting with the next frame ReShade can copy the contents of that texture/renderbuffer to its own texture that is then bound to the post-processing shaders.

There isn't really anything you can do to influence that (apart from making sure you use one of the API calls above to set up your FBO, but since those are all there exist as far as I know, this should be guaranteed). ReShade will always automatically search for the FBO that has most likely scene depth data in it. If it doesn't find any, it falls back to the main depth buffer. So yes, it does copy from the FBO as suggested.
It being upside down is not an issue, there is a preprocessor define on the settings page that tells all post-processing shaders to flip the depth texture before using it.

If you're just after MXAO, why not port it?

Thanks for the answer, crosire!

Marty McFly wrote: If you're just after MXAO, why not port it?

Forgive me for being totally code illiterate, but you mean by adding it to the plugin itself as an enhancement option? There was a request to add a number of post processing effects (including AO) to the plugin before, but the consensus on github seemed to be that it would be better to let an external program like ReShade handle post processing effects. Adding an option to GlideN64 to allow depth buffer access so that ReShade has better compatibility seems to be the more popular solution.

So MXAO is working- the shadows are applied upside down but setting the upside down preprocessor definition to "1" fixed that- but now the shading is still misaligned. The problem becomes worse when the game is in motion:


Gonetz says this is something that can be fixed by making Reshade use the depth buffer rather than FBO:

Yes, flipping FBO depth image will not be enough: FBO color image can be scaled or shifted somehow when rendered to screen. The same must be done for depth. The only way is to force Reshade to take main depth buffer as depth source. Maybe it is has some option for that? I can fill main depth buffer with correct data (hopefully).

As I understand it, right now GlideN64 doesn't copy depth buffer info to the main depth buffer, so the FBO is the only source ReShade can use. If he were to copy the depth buffer over, would ReShade automatically use that instead, or would it keep using FBO as a source?

It would keep using the FBO, because more draw calls were made with it. You can do any necessary scaling in the MXAO shader though.

I asked Marty but he said there wasn't a solution to this within MXAO... both he and Gonetz have suggested that it's up to Reshade at this point, so it seems like forcing Reshade to use the depth buffer instead of FBO would be the only resolution... is there any possibility of this being an option? If it's possible to do, it would allow a lot of effects to work with P64/GlideN64 that otherwise won't.

Sorry to bump this, but I wanted to get a clear answer one way or the other since Gonetz was very helpful and invested some of his time in trying to resolve this (thank you Crosire for your help as well). If the depth buffer were made available by GlideN64, would there be any way to specifically set Reshade to use it, or is Reshade just by nature totally unable to work without the current method of auto-selecting the depth source? I assume at the very least manually specifying the depth source would require a new version of Reshade with a number of changes, but even if it took a long while to be implemented I'd be happy to know it were possible.

It's not going to happen in the immediate future. Sorry.

Apparently this functions differently on Nvidia and AMD cards. With Nvidia, you can get depth based effects to work correctly in GlideN64 2.0 with framebuffer emulation disabled. On an AMD card, it's not possible to get the same result; you can get MXAO to apply in GlideN64 2.0, but it requires framebuffer emulation to be on and will be misaligned just as it is in the new versions of the plugin.

Gonetz speculated about what's causing that alignment issue I posted above (where MXAO is misaligned with the environment and moves 1 frame faster than the actual game, seen here where Link's shading swings before his model).


Again, MXAO worked correctly in the 2.0 version of the plugin on Nvidia cards and started this incorrect behavior in 3.0:
_________________
"GlideN64 2.0 uses main frame and depth buffers when framebuffer emulation is off.
Since PR 3.0 frame is always rendered with FBO regardless of framebuffer emulation mode.
Thus, main depth buffer is not used at all, so ReShade can't take information from it.

Then, rendered frame usually does not copied to main color buffer immediately after rendering. N64 games allocate at least two color buffers for double-buffering. One buffer is being rendered, other is displayed. Zelda (U) runs at 20 fps, while NTSC updates screen 60 times per second. That is ready frame will wait for 3 Video Interrupts before it will appear on screen. I guess, ReShade tries to use most recently updated FBO for MXAO, which explains why MXAO shading is a few frames faster than the actual game.

I don't know how ReShade works. May be it can take depth buffer from FBO, but it can be tricky because there are always several FBOs.
As an alternative, the plugin can copy not only color but also depth from FBO to main depth buffer. I may try to implement it"
_________________

He's willing to change a few things in the plugin if it can get depth effects working again, but he doesn't know enough about how ReShade works to know what will work. I'm also not sure if a depth buffer mod for OpenGL would work by allowing a different one of the buffers he mentioned to be selected. In the interest of helping him use his time effectively, is there any advice I could pass along to him?