(H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)

v2.9.1 is available, I made a few more improvements to the FXAA inline sharpening amount calculation and it's now able to handle much closer to 100% of what it gets fed without generating artifacts (probably 99.(some fraction)% at this point). Since the sharpening artifacts could sometimes look like an erroneous correction attempt on a real edge, the output of this version on true aliasing also looks slightly cleaner than 2.9.

Edit: v2.9.2 now up. Turns out a good chunk of the remaining artifacting problem wasn't a balancing issue, it was a logic issue. When I calculated maximum color separation for the pixel being processed I forgot to check contrast between green and blue, resulting in a hole in the results for certain color shades and thus an incorrect amount of sharpening being applied. It is now fixed and produces more consistent output.

Edit 2: v2.9.3 up. Now that I'm sure the inputs I'm using are being calculated correctly I rebuilt and retuned the final FXAA sharpen amount calculation. I don't think it generates artifacts any longer, spurious outputs on text seem to be coming from the anti-aliasing portion of FXAA at this point. The rebuilt calculation also adjusts itself better as the shader settings change since I used less constants than in the previous version.

Edit 3: Current version is up to 2.10.3. Implemented some new trickery with FXAA (it can be run on color pixels only, grayscale pixels only, or as a full pass) and the coarse passes added by DPI now start at 1080p, with passes alternating between color-only and grayscale-only. The sharpening calculation also got a tiny bit smarter again (it now considers how strongly the pixel cleared the edge detection threshold in the calculation).

Alrighty then, forget the v2.x strain ever existed. Version 3.0 is now available. Changelog:

Implemented witchcraft. Seriously, I don't know what I did. I was messing with the FXAA sharpening math WAY too late at night, tried something, and somehow ALL the blur disappeared. v3.0 has more aggressive presets, runs two fine FXAA passes by default, adds more of them as the DPI goes up (the coarse pass has been retired since it now seems irrelevant), and it somehow looks clearer than v2.x and even has a higher hit rate on real aliasing.

For those curious what I did since I've now managed to reverse engineer it being more awake - in my halfway brain-drunk overtired state I managed to change the interpolate in the FXAA final sharpening calculation to be between the original input pixel and the sharpened result instead of between the anti-aliased result and the sharpened result. In low luminance scenes this resulted in a total elimination of blur without any loss of anti-aliasing effect. Despite being tired at the time I still managed to set it up so that it works in high-luminance scenes too, albeit at the cost of a BIT of blur added back in.

Edit: there was still a lingering bug in the new sharpening process which I have now fixed and pushed as version 3.2.

Discovered and fixed a swathe of bugs stemming from how I handled the UI presets in 3.3 and overhauled + tuned the FXAA sharpening code in the current version, v3.3.2. Sharpening strength is now largely user-controlled (or preset-controlled) instead of calculated. This should allow for greater flexibility configuring HQAA to handle aliasing situations unique to different games. The Ultra preset now also acts as a sanity check for the code; the custom preset default values are identical to the Ultra preset, so toggling between Ultra and Custom should produce no visual change whatsoever (which, as far as I'm aware, is indeed the case). The difference between presets and Custom in older versions of the shader was being caused by aforementioned bugs.

v3.4 is now available on github. FXAA result sharpening can now either be manually controlled or placed in Automatic mode, wherein the pass will do some shenanigans with numbers to guess at an appropriate level of sharpening for the current pixel. Additionally implemented the ability to control the number of FXAA quality iterations in both the presets and manually, which opens up another method of setting a bias towards either performance or quality to users.

Implemented user and profile control over SMAA corner rounding in v3.4.1 instead of calculating it as it's largely a personal preference setting. Have also been working at keeping the UI clean, see example below.

v3.4.7 is now available. I advise anyone using a version that includes built-in presets to update to 3.4.7 to avoid encountering the glitch that I fixed. Versions with presets prior to 3.4.7 will cause ReShade and the game to crash when you enable Performance mode with the Custom preset selected.

v4.0 is now available. All remaining glitches from the v3 series (that I could find) have been fixed. v4 implements a more advanced edge detection method for both SMAA and FXAA which has radically increased the shader's ability to correctly identify and correct edges. I'd go as far as to say the output on ultra could pull its weight against a proper TAA implementation (at least in the anti-aliasing department - HQAA unfortunately can't stand in for TAA's other uses, like denoising).

Edit: v4.1 is now up, which revamps FXAA's logic used to select the primary luma color and rebalances the SMAA gamma bias applied to its luma weights. See below for a 1080p before/after.

Seeing how actively you fix, tweak and develop this, I'd like to give you one tip for the development process: Do have occasional longer pause from this and do something else instead.

I do audio engineering by trade and sound designing and producing music as a hobby. Out there it's well known fact that the faster you do, the better the quality is. The reasoning is that the longer you spend time listening what you're doing, the more you bias you hearing to find problems that actually are not there and it could easily spiral down to endless tweaking and fixing without ever being able to make a final product. When the process cannot be done fast, pauses are extremely important to "reset" your hearing and the bias of finding problems.

I have not doubt that the same would apply the shader coding when you look at the screen, tweaking the code and observing the changes. Too long time doing that and you'll surely bias your brains to find all kind of problems and places for tweaking and essentially become blind in search for perfection.

Just a friendly tip, nothing more.

On topic: The last version I've tested is 3.4.1. The one before that was something from the 2.x series. It seems apparent that the slight angles cannot be perfect on fullHD resolution without excessive blurring but it still seems to be much better on the later versions.

Regarding the slight angles, I've pretty much come to the same conclusion - it's simply not possible to achieve a perfect look once you pass a certain (small) ratio of texels to pixels. I've had to remember not to fall into the trap of trying to find a "perfect" configuration and instead to bear in mind to always try and prioritize general improvements over exact results.

As far as taking breaks, I'm probably starting into one as of v4.1.2. I feel with where the shader is at that I have optimized the output about as far as I can on a 1080p screen; I need to see results from my 1440p monitor before I can continue to make improvements.

With my previous post being said I still found a few ways to squeeze a bit more out of the current implementation 4.1 (which is now 4.1.5). I got my adaptive luma selection logic and specific weighting for SMAA calibrated better than I had it before, which allowed room for the defaults to be more aggressive due to SMAA reading less false positives.

Btw Tojkar, I don't recommend using a version lower than 3.4.7 long-term. There's a silent bug in the code in versions .4.7 that will crash the game if you enable performance mode with the custom preset selected. It took me a while to find it since I'd just been testing primarily using the Ultra preset, and Custom works fine in those older versions while in setup mode.

Version 4.3 is up, it's mostly unchanged from the last couple versions but includes a couple of new optional features. I put in an experimental denoiser which uses a sort of backwards FXAA (it detects edges below the threshold instead of above) to try and simulate TAA denoising, though I've honestly no idea whether this approach has any real merit, and I put in an optional full CAS pass which is almost a default CAS, with the added bonus of being able to detect where SMAA ran and reduce sharpening strength in those spots if the anti-aliasing result sharpening option is enabled.

Both of these new features are listed as separate effects in the ReShade list. The denoiser can be moved around freely in the list before/after other stuff (including HQAA), and the same is true of the CAS pass but with a quirk - if you place it above HQAA in the list, the SMAA detection part won't do anything.

v4.4 available. Denoiser option is disabled, it behaved like almost the opposite of a denoiser. I made the optional CAS considerably smarter in working with the core effect though, that seems to have been a good idea. I also removed the unconditional coarse pass (so at lower than 1440p the shader only runs one SMAA and one FXAA) but I also extended the max FXAA edge gradient search range considerably to compensate.

v6.0 is up. (I skipped posting the v5 version here.) Between the changes in the v5 update and v6, the shader now runs slightly slower, but the result accuracy is now at a nigh unbelievable level for something that isn't temporal. Example screenshot below which includes difficult lumas, oblique angles, false positive traps, some transparencies, and a fair amount of real aliasing. Ultra preset, sharpening also enabled @ default strength.



As a further example, here's a scene which is almost entirely a false positive (a menu).

Just a quick observation and question to satisfy my curiosity on the shader's inner workings.

I noticed that the blue parts in the Klegg tower are almost completely untouched even though they have very obvious aliasing. Why are those considered negatives by the shader?

The short answer is, because SMAA is really bad at detecting edges in pure blue scenes, and FXAA alone isn't enough to fix repeated-pattern aliasing.

The longer answer is because the first step of SMAA edge detection is to test the red and green color channels of each pixel looking for high-contrast areas, and then rendering to a hidden texture (which can be observed in the statistics tab) a red pixel for a probable vertical edge, or a green pixel for a probable horizontal edge (or if it thinks it's seeing both, it lights up both red and green on the texture, resulting in yellow). In all the experiments I've done with it, attempting to force it to use blue as the luminance color simply results in it missing edges everywhere - the less it relies on blue for luma, the more edges it detects. I tried to work around this as much as I could with my adaptive luma selecting methods in the first step, and it even takes a much stronger read from the gamma of the pixel (normally ignored, that's something else I did) if it detects a pure-blue scene, but overall results are going to be very hit and miss if there's no green or red for it to work with whatsoever.

Edit: that does give me an idea for a possible way forward though. It'll take me some time to experiment with and see if there's any merit to it, but there's some potential that I could have it jump horses mid-pass from luma detection to color detection if the adaptive luma process results in the fallback case of 3/8 blue, 5/8 gamma. It ought to be possible to use that as trigger condition to hook out to color edge detection and then just return the result from that routine.

Further edit: Chicken dinner for you, I believe. It's going to need more testing, but initial observations suggest this process works.

Tojkar, your curiosity has been one of the most useful things for the development of this shader. Every time you ask a question out of curiosity it seems to spark an idea in my head that results in improvement. Please, continue asking when you want to know something - the mental process of breaking something down into an explanation really seems to help analyze where the weak points in the system might be.

That goes for anyone else too, happy to explain (to the best of my ability) how or why I did something or why something happens the way it does with my shaders. Outright feedback also welcome, both positive and negative, as long as it's constructive. In the last two weeks the shader has gone from blurry and barely usable on 1080p to sharper than TAA with an anti-aliasing success rate almost on par with TAA at 1080p.

Please absolutely continue to point out where you can see the shader missing edges or generating odd results.

v7.0 is now available. It's a spicy little update that contains something for everyone - bug fixes, performance improvements, and quality enhancements, all in one package.

Man, revisions fly when you're having fun.

I upgraded from 4.1.5 to 7.3 and i get error when reshade is compiling:

14:04:07:381 [03896] | ERROR | Failed initializing "C:\Program Files (x86)\Origin Games\Dragon Age\bin_ship\reshade-shaders\Shaders\HQAA.fx":internal error: argument pulled into unrelated predicate

Same with for example 6.0.
I removed all the settings from reshade preset regarding hqaa.fx but still no success.

Dragon age origins is a dx9 32bit game.

Thanks for calling that out, klotim - I just tested it on Borderlands: the Pre-Sequel (also DX9 32bit) and got the same error, which means I've broken DirectX 9 support somehow. I'll troubleshoot it and report back with further news.