(H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)

Thanks!

New major version 16.0 now available. Replaced the previous linear normals data with a snapshot of pre-AA pixel lumas and used it to do hysterisis weighting calculation on FXAA final results. This has resulted in a fairly effective artifact minimization system, which has allowed SMAA corner rounding to be fully implemented into every preset except Potato.

16.1 now available, which fixes an issue that's probably been present without my knowledge for a while now (HQAA would fail to compile when used in an HDR title with the HDR output flag set to 0). It now compiles either way, but still needs the HDR output flag set to 1 in order to generate proper output in HDR games.

16.3 is now available. (16.2 was an HDR support update mostly focused on making sure everything worked in HDR modes.) The changes in 16.3 are nearly all performance optimizations, including a couple of ways I discovered I could squeeze a bit more speed out of SMAA (which surprised me, it's already a fairly well-optimized set of routines). Corner rounding usage in the quality presets is now no-holds-barred, GLaDOS turns it all the way up to max.

Edit: 16.3.1 is now current, which adds a pre-processor define to enable screenshot mode. Screenshot mode overrides the edge threshold to be 0.0 (regardless of preset OR custom setting) for purposes of taking screenshots. I don't recommend using it for normal gameplay as it will run VERY SLOWLY compared to normal.

Current version now 16.4. Subpixel effect strength bias removed from FXAA hysteresis calculation, which is instead simply multiplied by the precomputed gamma normal for the pixel. This results in improved correction of high-contrast edges where one side is especially dark without allowing any substantial amount of artifacting in the scene.

Screenshot montage time! Version 16.4 is visually in a pretty good place and I haven't posted comparisons in a while. All screenshots taken at 1440p with in-game anti-aliasing disabled and no other shaders loaded. HQAA is running core effect only (all pre-processor flags set to 0) and using the Ultra preset.

(screenshot montage continued)

(screenshot montage continued)

16.5 now available. Visuals are pretty much unchanged, however performance should be considerably better than 16.4. I found a few ways to do things more efficiently than before resulting in an overall faster shader. Although HQAA is still comparatively slow, the performance of the HQAA Ultra preset should be within the margin of error of running SMAA+FXAA separately (both with aggressive settings). YMMV, as this may also depend somewhat on the GPU in use.

It works great and smooth like butter in Rise of the Tomb Raider.
Thanks for your hard work!
Do you recommend High or Ultra settings?

Typically I would suggest using the least aggressive preset that manages to successfully correct the jaggies you're noticing - for most people, there isn't too much point in going above that. If you're the type that likes to take (and show off) screenshots, then running it a bit overkill makes more sense. I built the High and Ultra presets to target everyday use on lower-end and mid-range modern GPUs, respectively. GLaDOS is deliberately somewhat on the overkill side (though you can push the shader harder than that in Custom mode if you really want to).

Just playing Outer Wilds for the first time (I hear I'm in for a pretty good ride) but one thing I can say off the bat is HQAA is doing an absolutely bang-up job in this game. Here's a screenshot with HQAA Ultra (HQAACAS also enabled) followed by a low strength Deband.fx.

v17.0 now available. It represents a (mostly) final audit and cleanup of the code with the aim to make it a long-term release as I feel that the core effect is now working very well. There is one new feature though - instead of using a hardcoded calculation (since I've changed it so many times already) I set up the dynamic threshold adjustment range as a user/profile controlled option. Performance may also be very slightly improved, but the delta I measured is easily within run-to-run margin of error, so maybe it is, maybe it isn't.

Future development on the shader is likely to be limited to bug fixes or perhaps new optional features. I'm happy with the output quality from core HQAA (honestly, it significantly exceeded my expectations for what I could manage to make SMAA and FXAA do).

Edit: I did have one last major improvement in me, apparently. Version 17.1 "final boss edition" now available, which packs everything HQAA does into as few passes as possible and significantly reduces its total execution time.

Is it normal that HQAA has more performance impact then combo SMAA+FXAA?

@maciejsojka - Yes, for version 17.1...

Bet you thought that was the final release version, but nope... I present what is likely the FINAL final version, HQAA v17.2 "Turbo Boost Edition" (now available on github). Tested and confirmed to work with ReShade v5.0.0, and MUCH faster than 17.1.

I agree! HQAA 17.2 is much faster then 17.1! Thanks!

Minor update to 17.2.1 available. I improved the readability of the hysteresis debug mode and also the accuracy of the hysteresis calculation, which results in more detail from the original scene remaining after HQAA is done.

17.2.2 update now available. It fixes a couple of bugs with HQAA's HDR output mode.

17.2.3 update now available. Adds new optional fast debanding effect based on Deband.fx, with minor modifications for speed and HDR compatibility. Looks pretty nice in Diablo 3.

17.2.4 update now available. Changed the calculation done by the optional brightness booster to a new one that causes somewhat less contrast washout.

Deband as in dithering for the shader or deband for any visible banding caused by other shaders?
Does it make sense to add features that isnt really part of AA?