TAA-quality, non-blurry anti-aliasing done using SMAA+SmartSharpen (presets)

Several months ago, I discovered that TAA's ghosting side effect gives me rather bad headaches. Not satisfied with what FXAA or even basic SMAA could offer, I set about creating a preset of shader passes that can match, and sometimes even out-perform, the image quality level of TAA while operating solely on a frame-by-frame basis. Since there is no look-behind or look-ahead component to any of these passes, these presets do not cause ghosting. The final results linked here are the product of literal months spent staring at worst-case scenario scenes and making microscopic adjustments to each pass (thank you No Man's Sky for all your weird curved surfaces and never-quite-aligned camera angles).

WARNING - these are not performance-oriented presets. They will cut into your framerate visibly (I notice a typical drop of around 15-20% in performance, depending on game and situation). Although I have done my best to avoid it happening, you will also see occasional anti-aliasing artifacts (flickers or shimmers in the scene) due to the aggressive nature of the passes. However, the net result is still less prone to artifacts than TAA when it isn't sure it's seeing an edge. If you're really bothered by the specific look of SMAA anti-aliasing artifacts, and don't want to use TAA, then sorry, you need to look elsewhere.

This comes in either a two-pass or three-pass version. The basic concept that I have finalized is to split the jobs of detecting long edges, "everyday" edges and micro-aliasing into separate passes - however, for performance reasons, long edge and "everyday" edge were reasonably compatible to do in a single SMAA run. TL;DR version: both offer roughly the same visual quality, with the three-pass version costing more GPU time but producing less potential anti-aliasing artifacts than the two-pass version (since the detection and application of edge correction had to be made somewhat less specific to fit both in the same pass). I use the two-pass preset on the vast majority of my games.

For these to work, you will need the AstrayFX package in addition to the full default package and the one containing the SMAA shader. After installing it to a game, you will also need to navigate to the folder and create two copies of the file "SMAA.fx", naming them "SMAA2.fx" and "SMAA3.fx" (SMAA3 is only required if you're going to use the three-pass version). I also strongly advise disabling any driver-based sharpening or sharpening built into the game, since the smart sharpener in these presets is carefully configured to work with the SMAA passes for maximum image quality. Feel free to adjust the sharpening strength though, as it won't significantly affect image quality - it's the other things in the pass that were more important to match to SMAA.

Two-pass preset:  pastebin.com/qZr4KwFr
Three-pass preset:  pastebin.com/xnjxAYST

I may occasionally update those pastes if I discover an improvement that I can make; however at this point I have been working at it for quite a long time and am fairly convinced I have it tuned about as well as it can be.

This reply is meant to address the elephant in the room for anyone who's tried the presets and looked through the settings - yes, there's a good reason Luminance edge detection is used instead of Color edge detection. Selecting between the two was a point of contention for me for a long time; however I noticed that (while using the debug views) although Color edge detection appeared to detect more edges, the visual result was consistently not desirable. For some reason, Color edge detection causes these SMAA passes to become too aggressive with "maybe" edges, resulting in a lot more application of false positive anti-aliasing and hence increased levels of scene blur and anti-aliasing artifacts. The apparent drop in detection of edges in the Luminance detection code seems to simply represent a reduction in false positive edge detection and thus produces a cleaner image that is more true to the original (before post-processing) output.

You can still use other post process effects with these presets also, I forgot to include this part. I suggest running them before the SMAA-Sharpen-SMAA combinations for best image quality outcome (place them above my presets in the list). I personally like to inject the qUINT MXAO routine into a lot of older games. Looks way better than the default SSAO used in many older engines.

Does this forum have a time-out set for ability to edit posts? I was going to just post this as an edit to the main one, but the button is missing. I only get the option to reply or quote.

I just tried out both the 2-pass and 3-pass versions. Very uh... well, they require a lot of performance, that much is for sure.

As for the quality, stationary quality is very good, removes a lot, I wouldn't say it's non-blurry though. SMAA tends to do quite a bit of blurring.

I used FFXIV as my test subject since it has no truly good anti-aliasing setup,; what you offered works alright, aside from the text being mangled up, quality is nice.

I'd perhaps fine tune some of the settings within' SMAA on the edge detection. Using those small values means a lot is being detected, great, but kills performance. I know from when I used SMAA, you can really fine-tune those numbers so they detect what you want, without it going crazy on literally every single object and texture in a game, especially since SMAA can't just anti-alias everything it detects.

I personally still find FXAA's AA to be much more pleasant post-sharpened, but a combination of SMAA for depth detection and FXAA or a 2nd SMAA pass (that's not hyper aggressive) work for what they need to.

It's a nice job though and got me thinking about better ways to use what's available to reduce aliasing.

Yup, it's a pretty hoggish setup on GPU time. There's a reason why the super-fine pass is included in both presets though - it simulates TAA's round-off effect, wherein rogue white pixels get significantly damped or even eliminated entirely. I like the way this looks, my problem with TAA is simply (as I opened with) that its ghosting side effect consistently causes a headache for me. If you don't mind seeing the leftover white pixels from rasterized lighting, you can actually just turn off the final pass entirely in either preset.

Edit: Non-blurry is being used in a relative sense here. Running more than one SMAA pass is a recipe for a LOT of blur if you're not careful with the settings, and I think you'll find the result from my presets is about as non-blurred as it gets with the inclusion of the final "everything is an edge" blend pass. Honestly, I also find that even multiple SMAA passes cause less blur in texture details (non-geometry edges) than TAA tends to, which is why I call it a non-blurry setup (again, relatively speaking).

Yeah that's fair, TAA blurs more than just edges, as you said, textures and things lose very fine fidelity, something SMAA doesn't cause as much issue too.

Makes sense then, what you did and why you did it. Thanks for sharing it with the community!

Most welcome

Even if not that many people use it, I'm sure I'm not the only one out there who experiences trouble caused by ghosting or motion blur-esque effects, so perhaps some others will find this useful.

I was just thinking that the cheapest way to do with would be to use SMAA with motion blur -- possibly a custom-made motion blur shader that only goes 1 pixel distance. But then I realized that the motion blur would probably cause you headaches as well -- but it would only be a 1 pixel blur, so maybe not? 

Hmmm... how about this, then:  Instead of motion blur, a custom shader that detects when the camera moves (or its best effort anyways, using only the color buffer or depth buffer), and then (if the camera is moving) replaces the color pixels with one from a different mipmap level.  I'd imagine that would remove shimmering and sparklies and you probably wouldn't notice the image quality drop since the camera is moving.  I think there was a skyrim mod that did the same thing and people couldn't notice the image quality drop.  This might not work well due to problems with camera movement detection -- I think that skyrim mod was able to accurately determine camera movement speed through hooking.

 

I have half an idea that the same approximate results could be achieved using a modified SMAA that does the 1x1 blend right after the actual edge correction pass - that would cut the number of post process passes from 8 to 5 in the two-pass setup (not counting the sharpening) - it would probably introduce a little bit of error into the result since the 1x1 blend would apply based on detected edges before any anti-aliasing, but I suspect it would still do basically what I want it to. I don't know the programming well enough to try and implement this myself though. A big part of this also depends on where in the code the edge sensitivity is applied - this thought is only feasible if the sensitivity setting is calculated in the edge correction pass, since if it's in the edge detection pass (which honestly seems more likely), then I pretty much have to run the entire set of 4 again anyway.

Here's another version I've been working on with the intent of minimizing aberrations in the scene when comparing before-and-after images. It has an overall less aggressive look than the original two in my first post, however I find it still does a very good job of handling aliasing and additionally causes a lot less blur. The main changes consist of switching to color edge detection but applying it at a significantly reduced sensitivity, and changing smart sharpen into dumb sharpen at a low strength (this preset uses only the bilateral sharpen effect with no masking whatsoever since it seemed to play the nicest with this setup). Preset can be found here: pastebin.com/KJEQyt9h

I've updated the preset in the reply above with some modifications (added some corner rounding to handle oversize aliasing and changed the smart sharpening method again). Overall blur level was SLIGHTLY increased by the corner rounding, but it still seems to be causing less blur than the original two-pass preset, as well as hanging onto fine details better.

Aside - Seriously, why does the edit button keep disappearing? I hate forced multiposting like this.

Copied
Two-pass preset:  pastebin.com/qZr4KwFr
into SMAA2.fx and getting error.:

19:12:07:580 [04740] | ERROR | Failed to load "C:\Program Files (x86)\SquareEnix\FINAL FANTASY XIV - A Realm Reborn\game\reshade-shaders\Shaders\SMAA2.fx":
C:\Program Files (x86)\SquareEnix\FINAL FANTASY XIV - A Realm Reborn\game\reshade-shaders\Shaders\SMAA2.fx(1, 1): error X3000: syntax error: unexpected 'identifier'

EDIT.: nevermind, resolved it
EDIT.: using it for FFXIV, unfortunately fonts took a big hit in quality, but I guess that's expected.
Thanks anyway

 

lordbean wrote:

Aside - Seriously, why does the edit button keep disappearing? I hate forced multiposting like this.

Because spam bots were taking advantage. Posting content that looked harmless (copies of existing posts from years ago) and then after a while editing them with spam. Which was harder to detect.

Ah, that makes sense. I suppose it's better for everyone to live with occasionally making run-on posts than deal with actual spam.

use this uimask, you will need to edit it as shown in this linked issue

github.com/UTwelve/Reshade-shaders/issues/1

HackUIMask at top
HackUICut before effects
HackUIRestore after effects

Hello. I have some problem with TAA. Can this help ? I made a video to show my problem.  streamable.com/5nc693# (just the five first second)