AO Tweaks to deal with fog

As many people know, Ambient Occlusion is an effect that is generally applied to the entire image regardless of how bright it is. However, after finding that AO basically stencils on top of fog in games like Kotor 2, I am left wondering if there's a way to remove some of the AO based on how much Gamma is present in the scene. (Since the fog is essentially lighting up the scene so brightly that the AO is bleeding into it)
 
Basically I am proposing something like this     
into this  if possible.

I believe what you are looking for is a shader named Reveil. It comes as two shaders; Reveil top, Reveil Bottom. Place MXAO between them. Adjust Reveil settings as like:
Strength: -0.667
Depth Sensitivity: -1.000. (notice the minus sign)

What this does is, Reveil Top saves a picture of the frame. But, the closer the objects in the picture, more transparent they are saved. In the end there is a picture with visible details in greater distances and less visible (or non-visible) details in closer object.

Then MXAO does it's thing. Finally Reveil Bottom places the semi-transparent picture on top of MXAO'ed picture. That way, if MXAO is bleeding through a smoke, smoke is placed back on top of it and the depth order is preserved.

Thank you very much for the link.

I am having a problem with the shader you sent, though.. I can send this to the author's personal page, but I figure it's probably worth asking here first, (Not sure how Necros are treated on this Forum, so I'm being careful)

The Reveil shader is throwing an error. "error c7608 OpenGL does not allow using Discard in this profile" Not exactly well versed in how OpenGL works, but does this mean that the shader won't work on certain APIs, or rather is there a relatively easy way to fix it?

A Quick Update: Reveil now works perfectly with OpenGL due to the newest version of Reshade. You can find Reveil by activating Lord Of Lunacy's shaders while installing Reshade.

This shader fixed the issue and now it's much easier to make presets using AO without annoying artifacts in bright lighting and fog.

In my Glamayre_Fast_Effects shader, one of the effects is AO (it's a simple fast algorithm, not as pretty as MXAO).

Avoiding overshading foggy areas or transparent effects like smoke or explosions is difficult, but Glamayre has two techniques, both enabled by default. One based on depth, and one based on brightness. You can also tweak the threshold for both of them with two of the advanced settings:

**Reduce AO in bright areas** - Do not shade very light areas. Helps prevent unwanted shadows in bright transparent effects like smoke and fire, but also reduces them in solid white objects. Increase if you see shadows in white smoke. Decrease for more shade on light objects. Doesn't help with dark smoke.

**AO max distance** - The ambient occlusion effect fades until it is zero at this distance. Helps avoid avoid artefacts if the game uses fog or haze. If you see deep shadows in the clouds then reduce this. If the game has long, clear views then increase it.

Darker smoke or fog near the camera can still be a problem.

Long term, the ideal solution will be using ReShade 5's new plugin system to actually insert the AO shader earlier in the game's rendering pipeline. These two example plugins show it's possible:
github.com/crosire/reshade/tree/main/examples/02-shader_dump
github.com/crosire/reshade/tree/main/examples/03-shader_replace

Unfortunately, actually using them does not look very end-user friendly - they're really just examples for developers to build on. A nice GUI method for finding the place and injecting an AO shader would be nice.