Advanced Screen-space Subpixel Morphological Anti-Aliasing (ASSMAA)

More
4 weeks 20 hours ago #1 by lordbean
I've been screwing around with SMAA again... I present to you a very image-quality focused edit to the routine.

# ASSMAA

Advanced Screen-space Subpixel Morphological Anti-Aliasing is an edited version of traditional Subpixel Morphological Anti-Aliasing for ReShade 3.0+. It removes the optional depth edge detection and predication features of the original shader for best performance in multiplayer games and adds two advanced detection methods based on weighing the outputs of both Luminance and Color edge detection against each other in two different ways. ASSMAA also exposes the internal contrast adaptation setting to the user, allowing for finer control over the blurring caused as a side effect of the routine.

The two new detection modes offer up a way to use extra GPU horsepower to effectively interpolate the results of the outputs of both Color and Luminance edge detection. To be frank, I'm surprised this worked (I have a bad case of imposter syndrome right now) but during my testing in No Man's Sky (if you've played it you know it has tons upon tons of jaggies) I was able to identify subtle differences in the output weights of both new methods that identified them as distinct from either Color or Luma modes (or each other, for that matter). I've defaulted the options to a combination I've found to be very visually pleasing after some gameplay, but feel free to experiment - a lot of the idea of this edit is to give the user more power to fine-tune the shader.
The following user(s) said Thank You: canceralp, SpajdrEX, yoyas566, Judge_K

Please Log in or Create an account to join the conversation.

More
4 weeks 20 hours ago #2 by lordbean
Please note that if you have previously discovered and changed the SMAA_LOCAL_CONTRAST_ADAPTATION_FACTOR preprocessor definition and you then attempt to load this shader, you will receive an error while compiling it due to redefinition of SMAA_LOCAL_CONTRAST_ADAPTATION_FACTOR. The solution is to either manually edit the ReShade profile for the game and delete the definition from the start of the file, or to create a new profile for use with this shader (can be done right inside the ReShade UI).
The following user(s) said Thank You: someone_for, SpajdrEX, Judge_K

Please Log in or Create an account to join the conversation.

More
4 weeks 18 hours ago #3 by someone_for
Seems to be better than the usual SMAA. Very nice. Thank you!

Please Log in or Create an account to join the conversation.

More
2 weeks 2 days ago #4 by Judge_K
Thank you. I've really been enjoying this filter. Is it possible to port the edge detection method to other anti-aliasing filters (FXAA, NFAA, etc.)? 
On a side note, I still laugh about the the filter's name, particularly the acronym. Any ways, keep up the great work.

Please Log in or Create an account to join the conversation.

More
2 weeks 2 days ago - 2 weeks 2 days ago #5 by EFermi
What about morphological supersampling? As in, scale the frame buffer 2x2 or 4x4 in size, apply ASSMAA, scale down to initial. Should produce superb results due to interpolation (unless you choose to use point sampling / nearest neighbor for some ungodly reason). You may also add scaling filters choice of bicubic or lanczos. 
Last edit: 2 weeks 2 days ago by EFermi.

Please Log in or Create an account to join the conversation.

More
2 weeks 1 day ago #6 by Judge_K
As Crosire said, "ReShade applies shaders to the default back buffer. It does not and cannot change the resolution. As such, upscaling or down scaling is not really possible." reshade.me/forum/shader-suggestions/2313...-1080p-on-4k-monitor

A person could use something like DSR or VSR, which would apply the shader on the supersampled back buffer. Then the driver downscales the image. 



 

Please Log in or Create an account to join the conversation.

More
1 week 5 days ago - 1 week 5 days ago #7 by EFermi
Alternatively, is there a way to make SMAA sample the entire image just like FXAA does instead of just precisely targeting edges? 
The reason being that SMAA has much more accurate color mediation, but it results in not being able to process pixel-sized dots on transparent textures, like those tree leafs below, while FXAA does the job almost perfectly, if a bit mushy:

SMAA - imgur.com/hnhqzpt
FXAA - imgur.com/LMvsaDV

Unfortunately, FXAA has it's own serious demerit, huge enough to make me never use it and prefer no AA instead when only those two options (FXAA or SMAA) are available - inaccurate pixel inerpolation, which captures unwanted neighbouring pixels a lot, as an example see the farther table leg on images below (upscaled in linear interpolation for better visibility):

NoAA - imgur.com/rsF7Ns6
SMAA - imgur.com/mHwYBgD
FXAA - imgur.com/N5bmJvB

Notice how FXAA bleed surrounding pixels into the table leg on several instances. With more accurate color mediation of SMAA this should not be the problem. 
Last edit: 1 week 5 days ago by EFermi.

Please Log in or Create an account to join the conversation.

More
1 week 5 days ago - 1 week 5 days ago #8 by EFermi
Also, I've noticed that SMAA prefers to replace darker edge pixels with lighter ones (see same table leg SMAA vs NoAA) when mediating colors, instead of using 50/50 color data it seems to be 70/30 or smth. in favor of brighter shades, which is also a problem that made me notice a couple of very few, very-very subtle graphics artifacts at singular occasions. Would be great if this could be fixed. Obviosly SMAA shader misinterprets the dark and bright parts of the powerline like separate objects and due to it's imperfect handling does not mediate space between them, instead opting to separate the dark and the bright parts.

(View at 100% scale for better visualization)
NoAA - imgur.com/kS4yic4
SMAA - imgur.com/pCAW9kT
FXAA (never fails to disappoint, making even a simple multi-shaded line look like a mess) - imgur.com/pYZ0wkX
Postal 2 being Postal, even 4x supersampling with lanczos downscaling cannot be smooth, holy @$$ - imgur.com/mutzlIA

And, while we're on the topic, here's some more examples of foliage coverage.

FXAA (decent, but mushy) - imgur.com/effyVjY
SMAA (does it even do AA in that tree?) - imgur.com/O1QHRIE 

Fun thing is, even SMAA T2X cannot do foliage properly as I've witnessed in Crysis Remastered and Kingdom Come Deliverance - despite it being more generally blurry, it still misses those 1 pixel dots in the foliage among the tree leafs, it's just not as blatantly obvious as with normal SMAA.
 
Last edit: 1 week 5 days ago by EFermi.

Please Log in or Create an account to join the conversation.

More
1 week 5 days ago - 1 week 4 days ago #9 by EFermi
I can't do VSR because
a) it glitches the hell out in 7-8 out of 10 games putting my game into a borderless window 2/3 size of the monitor space, as if the monitor is above native, but the game still runs at it;
b) for some unexplainable reason AMD does not allow above 5K VSR for 4K mons, meaning I'm getting f**k all in terms of supersampling, getting from 3840x2160 up to 5120x2880, which is not even 1.5x supersampling (that would be 5760x3240), which looks arguably worse than native because the system struggles to interpolate whatever it has to work with resulting in "bad fxaa" kind of effect on edges when it sorta kinda has some semblance of an attempt at antialiasing, but fails miserably. FML. At least I can afford to not use AA and not suffer from it at my resolution.  

Regardless. An ideal antialiasing shader, in my opinion, should process within the textures similar to FXAA, but more cleanly, without capturing odd stray pixels and turning it all into mush, something you can see when using DSR with around 50% softness or very high levels of TSAA (transparency antialiasing). I am absolutely certain this is acievable with SMAA shader modified to unrecognisable state, probably, based on how cleanly it filters vast majority of edges compared to the original MLAA or FXAA, maybe by introducing some of the FXAA code into SMAA shader and creating an atrocious, but extremely capable hybrid of both.
This is what is expected to happen within the texture margins. 
Then, when shader reaches texture border, it works identical to old-fashioned SMAA with a small fix to balance the dark and bright pixel mediation shenanigans. 
Alternatively, the entire shader may be reconstructed to process the entire frame buffer regardless of edge detection, like I mentioned earlier, similar to FXAA, but far more smooth and accurate, without messing up edge pixels together, producing a significantly softened image which is then sharpened by other sharpening shaders like LumaSharpen and CAS.
Last edit: 1 week 4 days ago by EFermi.

Please Log in or Create an account to join the conversation.

More
1 week 4 days ago #10 by lordbean

Alternatively, is there a way to make SMAA sample the entire image just like FXAA does instead of just precisely targeting edges?
 

It actually does sample the entire image, interestingly enough. The first thing SMAA does is to basically copy the back buffer into a new texture, but as it does so, it runs a series of checks on the input to detect whether it thinks there's an edge there. If it detects an edge, it allows the data to be written to the sampled texture; if it doesn't, it simply drops the data and nothing shows on the sampler image. One of the things I've been looking at (but don't think I understand it well enough yet) is to try playing around with how it's actually determining whether or not it detects an edge.

Please Log in or Create an account to join the conversation.

More
1 week 4 days ago - 1 week 4 days ago #11 by EFermi
So how about removing this limitation completely, making every pixel an edge pixel? Will it turn into mushy mess like max. softness FXAA setting or will it just be blurred somewhat similarly to DSR at max. softness setting (more accurate gaussian blur)? If it's the latter it may be what I was looking for. 
Last edit: 1 week 4 days ago by EFermi.

Please Log in or Create an account to join the conversation.

More
1 day 3 hours ago #12 by lordbean
I've just made a significant update to the shader on github. When using detection method "Both, biasing anti-aliasing" the results are now computed using the geometric mean of the results of Luma and Color edge detection instead of a Boolean OR mathematical operation. This draws more information overall from either detection type and has a stronger dampening effect on rogue bright pixels than the previous version did. This is now the default mode used when first loading the shader. The clarity bias mode is unchanged as I feel it does its job very well using the current implementation of Boolean AND mathematical operation.
The following user(s) said Thank You: loki168, EFermi

Please Log in or Create an account to join the conversation.

More
9 hours 21 minutes ago #13 by EFermi
I'll definitely give it a try when I'm at home today.

Please Log in or Create an account to join the conversation.