Advanced Screen-space Subpixel Morphological Anti-Aliasing (ASSMAA)

I've been screwing around with SMAA again... I present to you a very image-quality focused edit to the routine.

# ASSMAA

Advanced Screen-space Subpixel Morphological Anti-Aliasing is an edited version of traditional Subpixel Morphological Anti-Aliasing for ReShade 3.0+. It removes the optional depth edge detection and predication features of the original shader for best performance in multiplayer games and adds two advanced detection methods based on weighing the outputs of both Luminance and Color edge detection against each other in two different ways. ASSMAA also exposes the internal contrast adaptation setting to the user, allowing for finer control over the blurring caused as a side effect of the routine.

The two new detection modes offer up a way to use extra GPU horsepower to effectively interpolate the results of the outputs of both Color and Luminance edge detection. To be frank, I'm surprised this worked (I have a bad case of imposter syndrome right now) but during my testing in No Man's Sky (if you've played it you know it has tons upon tons of jaggies) I was able to identify subtle differences in the output weights of both new methods that identified them as distinct from either Color or Luma modes (or each other, for that matter). I've defaulted the options to a combination I've found to be very visually pleasing after some gameplay, but feel free to experiment - a lot of the idea of this edit is to give the user more power to fine-tune the shader.

Please note that if you have previously discovered and changed the SMAA_LOCAL_CONTRAST_ADAPTATION_FACTOR preprocessor definition and you then attempt to load this shader, you will receive an error while compiling it due to redefinition of SMAA_LOCAL_CONTRAST_ADAPTATION_FACTOR. The solution is to either manually edit the ReShade profile for the game and delete the definition from the start of the file, or to create a new profile for use with this shader (can be done right inside the ReShade UI).

Seems to be better than the usual SMAA. Very nice. Thank you!

Thank you. I've really been enjoying this filter. Is it possible to port the edge detection method to other anti-aliasing filters (FXAA, NFAA, etc.)? 
On a side note, I still laugh about the the filter's name, particularly the acronym. Any ways, keep up the great work.

What about morphological supersampling? As in, scale the frame buffer 2x2 or 4x4 in size, apply ASSMAA, scale down to initial. Should produce superb results due to interpolation (unless you choose to use point sampling / nearest neighbor for some ungodly reason). You may also add scaling filters choice of bicubic or lanczos. 

As Crosire said, "ReShade applies shaders to the default back buffer. It does not and cannot change the resolution. As such, upscaling or down scaling is not really possible." reshade.me/forum/shader-suggestions/2313...-1080p-on-4k-monitor

A person could use something like DSR or VSR, which would apply the shader on the supersampled back buffer. Then the driver downscales the image. 



 

Alternatively, is there a way to make SMAA sample the entire image just like FXAA does instead of just precisely targeting edges? 
The reason being that SMAA has much more accurate color mediation, but it results in not being able to process pixel-sized dots on transparent textures, like those tree leafs below, while FXAA does the job almost perfectly, if a bit mushy:

SMAA - imgur.com/hnhqzpt
FXAA - imgur.com/LMvsaDV

Unfortunately, FXAA has it's own serious demerit, huge enough to make me never use it and prefer no AA instead when only those two options (FXAA or SMAA) are available - inaccurate pixel inerpolation, which captures unwanted neighbouring pixels a lot, as an example see the farther table leg on images below (upscaled in linear interpolation for better visibility):

NoAA - imgur.com/rsF7Ns6
SMAA - imgur.com/mHwYBgD
FXAA - imgur.com/N5bmJvB

Notice how FXAA bleed surrounding pixels into the table leg on several instances. With more accurate color mediation of SMAA this should not be the problem. 

Also, I've noticed that SMAA prefers to replace darker edge pixels with lighter ones (see same table leg SMAA vs NoAA) when mediating colors, instead of using 50/50 color data it seems to be 70/30 or smth. in favor of brighter shades, which is also a problem that made me notice a couple of very few, very-very subtle graphics artifacts at singular occasions. Would be great if this could be fixed. Obviosly SMAA shader misinterprets the dark and bright parts of the powerline like separate objects and due to it's imperfect handling does not mediate space between them, instead opting to separate the dark and the bright parts.

(View at 100% scale for better visualization)
NoAA - imgur.com/kS4yic4
SMAA - imgur.com/pCAW9kT
FXAA (never fails to disappoint, making even a simple multi-shaded line look like a mess) - imgur.com/pYZ0wkX
Postal 2 being Postal, even 4x supersampling with lanczos downscaling cannot be smooth, holy @$$ - imgur.com/mutzlIA

And, while we're on the topic, here's some more examples of foliage coverage.

FXAA (decent, but mushy) - imgur.com/effyVjY
SMAA (does it even do AA in that tree?) - imgur.com/O1QHRIE 

Fun thing is, even SMAA T2X cannot do foliage properly as I've witnessed in Crysis Remastered and Kingdom Come Deliverance - despite it being more generally blurry, it still misses those 1 pixel dots in the foliage among the tree leafs, it's just not as blatantly obvious as with normal SMAA.
 

I can't do VSR because
a) it glitches the hell out in 7-8 out of 10 games putting my game into a borderless window 2/3 size of the monitor space, as if the monitor is above native, but the game still runs at it;
b) for some unexplainable reason AMD does not allow above 5K VSR for 4K mons, meaning I'm getting f**k all in terms of supersampling, getting from 3840x2160 up to 5120x2880, which is not even 1.5x supersampling (that would be 5760x3240), which looks arguably worse than native because the system struggles to interpolate whatever it has to work with resulting in "bad fxaa" kind of effect on edges when it sorta kinda has some semblance of an attempt at antialiasing, but fails miserably. FML. At least I can afford to not use AA and not suffer from it at my resolution.  

Regardless. An ideal antialiasing shader, in my opinion, should process within the textures similar to FXAA, but more cleanly, without capturing odd stray pixels and turning it all into mush, something you can see when using DSR with around 50% softness or very high levels of TSAA (transparency antialiasing). I am absolutely certain this is acievable with SMAA shader modified to unrecognisable state, probably, based on how cleanly it filters vast majority of edges compared to the original MLAA or FXAA, maybe by introducing some of the FXAA code into SMAA shader and creating an atrocious, but extremely capable hybrid of both.
This is what is expected to happen within the texture margins. 
Then, when shader reaches texture border, it works identical to old-fashioned SMAA with a small fix to balance the dark and bright pixel mediation shenanigans. 
Alternatively, the entire shader may be reconstructed to process the entire frame buffer regardless of edge detection, like I mentioned earlier, similar to FXAA, but far more smooth and accurate, without messing up edge pixels together, producing a significantly softened image which is then sharpened by other sharpening shaders like LumaSharpen and CAS.

EFermi wrote:

Alternatively, is there a way to make SMAA sample the entire image just like FXAA does instead of just precisely targeting edges?
 

It actually does sample the entire image, interestingly enough. The first thing SMAA does is to basically copy the back buffer into a new texture, but as it does so, it runs a series of checks on the input to detect whether it thinks there's an edge there. If it detects an edge, it allows the data to be written to the sampled texture; if it doesn't, it simply drops the data and nothing shows on the sampler image. One of the things I've been looking at (but don't think I understand it well enough yet) is to try playing around with how it's actually determining whether or not it detects an edge.

So how about removing this limitation completely, making every pixel an edge pixel? Will it turn into mushy mess like max. softness FXAA setting or will it just be blurred somewhat similarly to DSR at max. softness setting (more accurate gaussian blur)? If it's the latter it may be what I was looking for. 

I've just made a significant update to the shader on github. When using detection method "Both, biasing anti-aliasing" the results are now computed using the geometric mean of the results of Luma and Color edge detection instead of a Boolean OR mathematical operation. This draws more information overall from either detection type and has a stronger dampening effect on rogue bright pixels than the previous version did. This is now the default mode used when first loading the shader. The clarity bias mode is unchanged as I feel it does its job very well using the current implementation of Boolean AND mathematical operation.

I'll definitely give it a try when I'm at home today.

Nah, same stuff essentially. Tested on Crysis.

It's a subtle change, not a major one. Keep in mind this is still just the normal SMAA routines at its heart, as I don't even include the header file with the project - it uses the same underlying code.

my apology, i havent noticed an interesting difference between this shader and classical SMAA Color detection + depth detection at 1080P.  I would even say that it reduces the quality by removing the depth.
i always used smaa at medium values as a prepass and depth detection is particularly effective in this case as it reduces the amount of bluriness by telling the shaders which parts of the picture to avoid. Adding luma to the mix may add a lot of ignored edges thoo...

The interpolation idea is great but i dont undestand why you remove the depth detection.
We can also access depth in online game by forcing it with forks or a "trick" in old versions of reshade ( as it is not intended i wont talk further about it). by doing this, smaa quality and bluriness is definitively improved.. To reduce the quality of a shader to adapt it for online games, it is a bit strange considering the goal of improving the picture quality with more horse power.

note that i use particularly tweaked values for smaa, with low amount of search steps

[SMAA.fx]
PredicationEnabled=4294967295
CornerRounding=0
DepthEdgeDetectionThreshold=0.003000
DebugOutput=0
EdgeDetectionThreshold=0.083333
EdgeDetectionType=1
MaxSearchSteps=16
PredicationScale=1.000000
MaxSearchStepsDiagonal=4
PredicationStrength=0.500000
PredicationThreshold=0.001667

such setting is an efficient prepass for the rests of the effects in the shader chain. then i add sharpen and dark blur aimed at bright pixels that smaa always misses (or very light fxaa avoiding dark pixels if the remaining aliasing is too high)

Without diving too deeply into the guts of the code, I can tell you there was a definite secondary reason I disabled depth edge detection. Predication is actually integrated into the detection methods in the header file, such that when you tick the "enable predication" box, you're actually changing to two new detection modes (Predicated Color or Predicated Luma) which means there's no easy way to integrate it into the color+luma interpolation methods I came up with unless I start making changes to the header file.

As far as your comments about not noticing a difference, that honestly isn't surprising. Comparing final images of traditional detection vs my new methods, I generally can't see a difference either. It takes a good amount of time spent scrutinizing the debug output layers in each mode to learn where the differences actually are in order to train your eyes to see them in the final product image. The differences are typically quite subtle and mostly center around how the shader handles difficult cases (where it isn't quite sure if it's really seeing an edge or not).

Essentially, I haven't re-invented the wheel. I've just done my best to try to grind it down into a more perfect circle.