TAA-quality, non-blurry anti-aliasing done using SMAA+SmartSharpen (presets)

4 days 21 hours ago - 4 days 21 hours ago #1 by lordbean
Several months ago, I discovered that TAA's ghosting side effect gives me rather bad headaches. Not satisfied with what FXAA or even basic SMAA could offer, I set about creating a preset of shader passes that can match, and sometimes even out-perform, the image quality level of TAA while operating solely on a frame-by-frame basis. Since there is no look-behind or look-ahead component to any of these passes, these presets do not cause ghosting. The final results linked here are the product of literal months spent staring at worst-case scenario scenes and making microscopic adjustments to each pass (thank you No Man's Sky for all your weird curved surfaces and never-quite-aligned camera angles).

WARNING - these are not performance-oriented presets. They will cut into your framerate visibly (I notice a typical drop of around 15-20% in performance, depending on game and situation). Although I have done my best to avoid it happening, you will also see occasional anti-aliasing artifacts (flickers or shimmers in the scene) due to the aggressive nature of the passes. However, the net result is still less prone to artifacts than TAA when it isn't sure it's seeing an edge. If you're really bothered by the specific look of SMAA anti-aliasing artifacts, and don't want to use TAA, then sorry, you need to look elsewhere.

This comes in either a two-pass or three-pass version. The basic concept that I have finalized is to split the jobs of detecting long edges, "everyday" edges and micro-aliasing into separate passes - however, for performance reasons, long edge and "everyday" edge were reasonably compatible to do in a single SMAA run. TL;DR version: both offer roughly the same visual quality, with the three-pass version costing more GPU time but producing less potential anti-aliasing artifacts than the two-pass version (since the detection and application of edge correction had to be made somewhat less specific to fit both in the same pass). I use the two-pass preset on the vast majority of my games.

For these to work, you will need the AstrayFX package in addition to the full default package and the one containing the SMAA shader. After installing it to a game, you will also need to navigate to the folder and create two copies of the file "SMAA.fx", naming them "SMAA2.fx" and "SMAA3.fx" (SMAA3 is only required if you're going to use the three-pass version). I also strongly advise disabling any driver-based sharpening or sharpening built into the game, since the smart sharpener in these presets is carefully configured to work with the SMAA passes for maximum image quality. Feel free to adjust the sharpening strength though, as it won't significantly affect image quality - it's the other things in the pass that were more important to match to SMAA.

Two-pass preset:  pastebin.com/qZr4KwFr
Three-pass preset:  pastebin.com/xnjxAYST

I may occasionally update those pastes if I discover an improvement that I can make; however at this point I have been working at it for quite a long time and am fairly convinced I have it tuned about as well as it can be.
Last edit: 4 days 21 hours ago by lordbean. Reason: No auto-link creation - manually created links to pastebin dumps
The following user(s) said Thank You: anontsuki

Please Log in or Create an account to join the conversation.

4 days 8 hours ago #2 by lordbean
This reply is meant to address the elephant in the room for anyone who's tried the presets and looked through the settings - yes, there's a good reason Luminance edge detection is used instead of Color edge detection. Selecting between the two was a point of contention for me for a long time; however I noticed that (while using the debug views) although Color edge detection appeared to detect more edges, the visual result was consistently not desirable. For some reason, Color edge detection causes these SMAA passes to become too aggressive with "maybe" edges, resulting in a lot more application of false positive anti-aliasing and hence increased levels of scene blur and anti-aliasing artifacts. The apparent drop in detection of edges in the Luminance detection code seems to simply represent a reduction in false positive edge detection and thus produces a cleaner image that is more true to the original (before post-processing) output.

Please Log in or Create an account to join the conversation.

2 days 13 hours ago #3 by lordbean
You can still use other post process effects with these presets also, I forgot to include this part. I suggest running them before the SMAA-Sharpen-SMAA combinations for best image quality outcome (place them above my presets in the list). I personally like to inject the qUINT MXAO routine into a lot of older games. Looks way better than the default SSAO used in many older engines.

Does this forum have a time-out set for ability to edit posts? I was going to just post this as an edit to the main one, but the button is missing. I only get the option to reply or quote.

Please Log in or Create an account to join the conversation.

2 days 10 hours ago #4 by anontsuki
I just tried out both the 2-pass and 3-pass versions. Very uh... well, they require a lot of performance, that much is for sure.

As for the quality, stationary quality is very good, removes a lot, I wouldn't say it's non-blurry though. SMAA tends to do quite a bit of blurring.

I used FFXIV as my test subject since it has no truly good anti-aliasing setup,; what you offered works alright, aside from the text being mangled up, quality is nice.

I'd perhaps fine tune some of the settings within' SMAA on the edge detection. Using those small values means a lot is being detected, great, but kills performance. I know from when I used SMAA, you can really fine-tune those numbers so they detect what you want, without it going crazy on literally every single object and texture in a game, especially since SMAA can't just anti-alias everything it detects.

I personally still find FXAA's AA to be much more pleasant post-sharpened, but a combination of SMAA for depth detection and FXAA or a 2nd SMAA pass (that's not hyper aggressive) work for what they need to.

It's a nice job though and got me thinking about better ways to use what's available to reduce aliasing.

Please Log in or Create an account to join the conversation.

2 days 8 hours ago - 2 days 8 hours ago #5 by lordbean
Yup, it's a pretty hoggish setup on GPU time. There's a reason why the super-fine pass is included in both presets though - it simulates TAA's round-off effect, wherein rogue white pixels get significantly damped or even eliminated entirely. I like the way this looks, my problem with TAA is simply (as I opened with) that its ghosting side effect consistently causes a headache for me. If you don't mind seeing the leftover white pixels from rasterized lighting, you can actually just turn off the final pass entirely in either preset.

Edit: Non-blurry is being used in a relative sense here. Running more than one SMAA pass is a recipe for a LOT of blur if you're not careful with the settings, and I think you'll find the result from my presets is about as non-blurred as it gets with the inclusion of the final "everything is an edge" blend pass. Honestly, I also find that even multiple SMAA passes cause less blur in texture details (non-geometry edges) than TAA tends to, which is why I call it a non-blurry setup (again, relatively speaking).
Last edit: 2 days 8 hours ago by lordbean.

Please Log in or Create an account to join the conversation.

2 days 8 hours ago #6 by anontsuki
Yeah that's fair, TAA blurs more than just edges, as you said, textures and things lose very fine fidelity, something SMAA doesn't cause as much issue too.

Makes sense then, what you did and why you did it. Thanks for sharing it with the community!

Please Log in or Create an account to join the conversation.

2 days 7 hours ago #7 by lordbean
Most welcome :)

Even if not that many people use it, I'm sure I'm not the only one out there who experiences trouble caused by ghosting or motion blur-esque effects, so perhaps some others will find this useful.

Please Log in or Create an account to join the conversation.