Marty McFly's Ambient Obscurance (MXAO) with IL

More
4 years 8 months ago - 4 years 8 months ago #401 by Nerrel
Really happy to see 3.0 release, but I'm getting an error that reads "warning X3206: Implicit truncation of vector type" followed by a list of errors when running this with Project64. MXAO 2.0 just worked perfectly fine with everything set up exactly the same... hopefully it's something that can be resolved.

Screengrab of error: c1.staticflickr.com/5/4473/37096546043_074c51c251_o.jpg
Last edit: 4 years 8 months ago by Nerrel.
The topic has been locked.
More
4 years 8 months ago #402 by Marty McFly
Replied by Marty McFly on topic Marty McFly's Ambient Obscurance (MXAO) with IL
ShoterXX: Whoopsie, it was late in the evening when I did that...
Nerrel: that's really odd. Is Project64 OpenGL? I ran MXAO just fine on both DX9 and 11 so this sounds like a ReShade bug, if it's really that.
The topic has been locked.
More
4 years 8 months ago #403 by GP-Unity
This is working really well now :D doing a lot of experimenting but great results so far. I adjust 'float3 lumcoeff = float3(0.2126, 0.7152, 0.0722);' if i want to refine colours a bit depending on game right?
The topic has been locked.
More
4 years 8 months ago #404 by ShoterXX
Nah, you're not supposed to mess with that. That's the brightness that each color channel emits, used to calculate light emitted.
The topic has been locked.
More
4 years 8 months ago - 4 years 8 months ago #405 by Marty McFly
Replied by Marty McFly on topic Marty McFly's Ambient Obscurance (MXAO) with IL
Correct. If you want to calculate Luma, you could just average all 3 channels. As eye is most sensitive to green, a pure green pixel appears brighter than a pure blue or red one. These coefficients you see weight red green and blue in such a way that the pixels with same perceived brightness have the same calculated Luma value.

@everyone: would you be up for more blending modes? As there "can't" be a perfect one, I'd just add a few that come to my mind as just enforcing one that I view as best is inconsistent.

@Nerrel: forgot to say, this is really critical and I can't push an update on MXAO as long as this is around. I'll try to find an OpenGL app that works with ReShade...
Last edit: 4 years 8 months ago by Marty McFly.
The topic has been locked.
More
4 years 8 months ago #406 by GP-Unity
More blending modes would be awesome :)
The topic has been locked.
More
4 years 8 months ago - 4 years 8 months ago #407 by Nerrel

ShoterXX: Whoopsie, it was late in the evening when I did that...
Nerrel: that's really odd. Is Project64 OpenGL? I ran MXAO just fine on both DX9 and 11 so this sounds like a ReShade bug, if it's really that.


It depends on the graphics plugin. GlideN64 (the best plugin for enhancement in my opinion) is OpenGL, and it worked perfectly with MXAO 2.0 but required disabling framebuffer emulation to get depth buffer access... I might try to completely reinstall REshade using the newest version and MXAO 3.0 and see if that changes anything.

EDIT: Yep, same result after setting it up again. If it's useful, I can send you the Reshade log with the entire error string in it.
Last edit: 4 years 8 months ago by Nerrel.
The topic has been locked.
More
4 years 8 months ago - 4 years 8 months ago #408 by Marty McFly
Replied by Marty McFly on topic Marty McFly's Ambient Obscurance (MXAO) with IL
Okay, here's a new version:

pastebin.com/FPJGrvrb

Changes are a) 2 more blending modes! and b) I have a hunch what went wrong on Nerrel's end - got no OpenGL app to test. Attempted to workaround this, wouldn't be the first time for MXAO to break ReShade or the driver... If it still breaks, I need the log.

The blending modes: 0 is standard, 1 is what I made earlier, with a slight tweak (or none, didn't check) and 2 is another attempt at a more realistic and sophisticated blending mode. Mode 1 has the culprit that white IL onto objects = whiteish spot instead of object being brighter.
Last edit: 4 years 8 months ago by Marty McFly.
The following user(s) said Thank You: GP-Unity, Nerrel
The topic has been locked.
More
4 years 8 months ago #409 by ShoterXX
DX9 doesn't seem to enjoy the switch-case.
Other than that, blend mode #2 seems to make IL a lot more meaningful and nicer looking overall.
The topic has been locked.
More
4 years 8 months ago - 4 years 8 months ago #410 by Nerrel
So this time I get "Error C5215: Integer varying _param_MXAO_samples must be flat." I'm not sure how to post the log without overtaking the whole page, so I hope mega is Ok:
mega.nz/#!PNMRgRrT!OuEQ2KuP1R-11VWOHCEzKT2Z-lhW0bhlMDIvM4Okbmw
Last edit: 4 years 8 months ago by Nerrel.
The topic has been locked.
More
4 years 8 months ago #411 by Marty McFly
Replied by Marty McFly on topic Marty McFly's Ambient Obscurance (MXAO) with IL
Can't look at the entire log right now as I'm on the phone. This however indicates that we have a major problem:

www.opengl.org/discussion_boards/showthr...x-to-fragment-shader

Last post. I can't directly steer openGL code with ReShade.
The topic has been locked.
More
4 years 8 months ago #412 by GP-Unity

The blending modes: 0 is standard, 1 is what I made earlier, with a slight tweak (or none, didn't check) and 2 is another attempt at a more realistic and sophisticated blending mode. Mode 1 has the culprit that white IL onto objects = whiteish spot instead of object being brighter.


I've spent a couple of hours testing out the different blending modes, and i'm surprised i like mode 1 the most despite its' whiteish' colouring. Might be because it has more lighting coverage. Each blend mode is really great though, i'm sure the best choice varies per game which i'll have to spend a lot of time testing.
The topic has been locked.
More
4 years 8 months ago #413 by crosire

Last post. I can't directly steer openGL code with ReShade.

Change
struct MXAO_VSOUT
{
    float4 position : SV_Position;
    float2 texcoord : TEXCOORD0;
    float2 scaledcoord : TEXCOORD1;
    int samples : TEXCOORD2;
    int blursamples : TEXCOORD3;
    float3 uvtoviewADD : TEXCOORD4;
    float3 uvtoviewMUL : TEXCOORD5;
};
to
struct MXAO_VSOUT
{
    float4 position : SV_Position;
    float2 texcoord : TEXCOORD0;
    float2 scaledcoord : TEXCOORD1;
    nointerpolation int samples : TEXCOORD2;
    nointerpolation int blursamples : TEXCOORD3;
    float3 uvtoviewADD : TEXCOORD4;
    float3 uvtoviewMUL : TEXCOORD5;
};
The topic has been locked.
More
4 years 8 months ago #414 by Marty McFly
Replied by Marty McFly on topic Marty McFly's Ambient Obscurance (MXAO) with IL
Oh, I didn't know that such thing exists. So noninterpolation is the HLSL/ReShade FX pendant to flat in GLSL?
The topic has been locked.
More
4 years 8 months ago #415 by crosire
Correct.
The topic has been locked.
More
4 years 8 months ago - 4 years 8 months ago #416 by Marty McFly
Replied by Marty McFly on topic Marty McFly's Ambient Obscurance (MXAO) with IL
Learn something new every day...thank you for stepping in.
Alright, so I added crosire's fix and replaced that switch to make the warnings go away on d3d9. Nerrel, please report if it works for you. I raised the version number to 3.1.0 to avoid confusion with ReShade itself.

pastebin.com/Btyfrq2U

scratch that, one of my beta testers says it still doesn't compile due to same error
Last edit: 4 years 8 months ago by Marty McFly.
The topic has been locked.
More
4 years 8 months ago #417 by Nerrel
Yeah, same here. This one says Error C7561: OpenGL requires 'in/out/ with 'flat'.

What would cause this version to have OpenGL problems where 2.0 didn't?
The topic has been locked.
More
4 years 8 months ago #418 by Marty McFly
Replied by Marty McFly on topic Marty McFly's Ambient Obscurance (MXAO) with IL
3.0 has huge restructurements. The issue here is that I calculate some static but costy stuff in vertex shader so it won't be calculated for every pixel and pass it per to the pixel shader along with texcoord and the usual stuff. Here, integers seem to cause the problem. As Matsilagi provided me with an OpenGL game, I could find a fix which is really really stupid but works, switching the two integers to floats works...

pastebin.com/Gn3EUMyf
The following user(s) said Thank You: Nerrel
The topic has been locked.
More
4 years 8 months ago - 4 years 8 months ago #419 by crosire

Yeah, same here. This one says Error C7561: OpenGL requires 'in/out/ with 'flat'.

That's a bug in ReShade that has already been fixed in source code a while ago, but not released as a new version yet.
Last edit: 4 years 8 months ago by crosire.
The topic has been locked.
More
4 years 8 months ago #420 by Nerrel
Yep, it's working now. It seems the sample radius setting is very different from 2.0; in order to match the visuals with a radius of around 3 in the old version I had to set to 9 in this version. It definitely seems sharper and more detailed now, like on the pegs on the rack on the left (2.0 on left, 3.1 on right).

File Attachment:
File Attachment:
The topic has been locked.