(H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)

High quality modes misinterpret edge pixels on some very rare occasions.
Example from FEAR:

imgur.com/FyhQamN - default "High" quality setting or higher results in brighter pixel inrusion onto the darker area.

imgur.com/HSe6Hc2 - default "Low" quality setting does not have this error.

SMAA tends to create similar distortions if the settings are cranked high up (specifically diagonal edge detection), which is different from FXAA mad random pixel mashing, instead, it looks like SMAA-based code, including HQAA, genuinely has trouble differentiating some seemingly straight edges if they are "grainy", like the surface of this vent duct in the image, for example.
 

Yup, those are largely compound effects coming from slight errors from SMAA being multiplied by FXAA (which subsequently thinks it's a form of aliasing). Try turning up the Hysteresis strength as high as you can tolerate (generally it starts to look like it's counteracting the anti-aliasing at some point). It won't remove the problem but it'll make it less visually obvious.

v21.2 now available. I reworked the dynamic thresholding calculations to use the computed saturation level of the pixel instead of the luminance (this allows HQAA to use a lowered threshold for low-contrast dots at any luminance level) and additionally changed the error reduction maximum allowable neighbors to a user+profile controlled option as changing it can be used to either preserve detail or increase AA effect.

v21.3 now available. SMAA edge detection now uses location-appropriate subpixel weights instead of fixed ones, which gives it a much more complete and accurate analysis of the edges in the scene (this doesn't work for FXAA because FXAA doesn't perform subpixel corrections).

v21.4 now up. This version reworks dynamic thresholding to calculate based on both the luma and saturation of the pixel making it more flexible in different situations. Lots of profile and default settings ended up being changed to fit the new edge detection results. Overall this version should be noticeably more accurate (in particular it should have significantly fewer cases of missed edges) as compared to previous versions.

v22.0 now available. HQAA now performs an adaptive denoising pass after the completion of SMAA blending and another after all FXAA samples have completed. This pass uses SMAA edge data to perform denoising of edge gradients as accurately as possible, and overall this version produces a slightly softer-looking final image as compared to the previous one.

Edit: updated to v22.1, which overhauls and streamlines the UI for simplicity. All primary effect options are now either constants or controlled by preset selection for user friendliness. When in advanced mode (which replaces the previous custom preset and is activated via a preprocessor define), all parts of the primary effect become user configurable including a new option controlling the strength of the denoising effect.

Edit 2: updated to v22.2, which is a full-shader sweep for optimizations and bug fixes. I'm unsure whether it runs faster than previous versions or not, but it seems to compile to a considerably fewer number of instructions per pixel shader than it did before - at the very least, that's a plus for compatibility (it's possible to overflow the instruction buffers on old or cheap GPUs).

Derp: imgur.com/xEREZj2

Happens with both RS 4.9.1 and 5.0.2; game - F.E.A.R. (DX9 render, maybe that's the problem?).

Yep, seems to be it. Enables just fine on DooM Eternal but fails at both FEAR and its addon Extraction Point. Will try a few moar games.

Alan Wake (DX9) - fail.
Alan Wake Remastered (DX12) - functional.
Crysis (DX10) - functional.
Seems like DX9 problem?

Also failed on Darkest Dungeon (OGL).

Seems like it doesn't go well with older APIs. Version 20 worked fine.

1909 divide by zero... let's see if I can find out why that might be.

Edit: Under the assumption you were using the Custom profile with the previous version, I can see a path to a divide by zero in the code which should be impossible to reproduce when the settings are within the new "correct" range (this might cause the compiler to throw an error even if you have enabled the new advanced mode). If you edit the ReshadeProfile.ini file for the game and change HqaaPreset to any integer between 0 and 3, it should compile.

Edit 2: apparently not. I'm getting the same error thrown in Borderlands TPS on a clean profile. Appears to be a symptom of something underlying - I don't see a reason for a divide by zero to be thrown there in the code. I'll update when I get it fixed.

v22.2.1 now available, which should correct the problem for both D3D9 and OpenGL. This is a weird one as I'm honestly not sure what triggered the compile failure - the culprit appears to be using an array-indexed value as the denominator in a division, but HQAA was actually doing that prior to this version without a problem. Whatever's going on, HQAA now changes its approach in the affected code sections if the renderer is D3D9 or OpenGL to avoid the compiler error.

Updated to 22.3 on github. This is a quality of life update mostly affecting shader setup when not using advanced mode, and particularly aims to reduce confusion of configuring optional effects.

v23.0 now available. In addition to luma, SMAA edge detection now also checks chroma differences between relevant pixels and uses whichever result returns a stronger read. This makes it more able to see edges in areas of similar luminance with varying colors.

Aaaand just like that v23 is obsolete. v24.0 now available, which is the result of smacking v23.0 over the head with a rather large optimization hammer. Removed a bunch of passes that had little to no beneficial effect, found an optimization that dropped another full pass, and retuned all defaults and profiles to take full advantage of the new hybrid luma+chroma edge detection method in v23. You should see significantly better performance out of version 24 and possibly even higher quality due to better calibration of the profiles.

I seem to be more or less on fire today. v25.0 now available, which adds a new anti-shimmering feature: edge detection temporal aggregation. In order to cover for spurious missed edges during movement, HQAA now aggregates detected edges over two frames. This makes it able to "assume" an edge is still present even if it momentarily fails to detect it. Performance cost and resource overhead are both relatively light as I was able to largely incorporate this idea into existing defined textures; however one new texture was required to save frame n-1 edge data.

v25.2 now available (v25.1 reorganized SMAA edge detection in a way that makes better use of the dual detection methods). 25.2 expands on the edge detection temporal aggregation feature by using data from three frames instead of two. This is accomplished by simply using the other two color channels available in the already-declared new texture, and so additional overhead is very minimal.

v25.3 now available. I reworked the Chroma edge detection path to a system that can much more reliably detect edges where Luma detection typically fails. This necessitated a slight rework of the dynamic edge threshold reduction calculation to compress the range in which a strong reduction will be applied as the hybrid Luma + Chroma system now much more reliably generates good data without the extra help. The quality presets have also been reworked to good values for the new code.

I suspect this version will likely be a longer-term (for me, anyway) release as its output appears to be exceptionally good. As usual please report any issues with the shader so that I can troubleshoot and fix them.

v25.3.1 available, which is more or less just a further fine-tuned version of 25.3, although there could be a slight quality increase in the output because I found a small bug in the way chroma edge detection was applying local contrast adaptation. For those using Advanced mode setup, detected edges error reduction can now be skipped entirely (instead of being restricted to low/medium/high) by using the new "Off" bullet (although I recommend leaving a high margin of error at minimum as it seems to substantially increase the overall quality of results in my experience).

Well folks, it seems I lied.

v26.0 is now available. FXAA is now capable of dynamically selecting either luma or chroma edge detection in addition to SMAA. In my defense, I didn't actually think what I did was going to work - but then it did, so here we are. Also made a couple of optimizations and renamed a few internals to be more representative of what the code is using them for.

Edit: Current version is now 26.2, which has substantially improved edge detection accuracy for both the chroma and luma paths as compared to 26.0.

Here's a quick demo screenshot for 26.2.

v26.3 is now available. In addition to a considerable shader-wide performance improvement, I have done some spring cleaning in the optional effect subcategories (text blurbs are gone and the information has moved to tooltips in appropriate locations) and considerably improved the temporal stabilizer effect with addition of a new edges-only option and a smarter max weight clamp calculation.

lordbean,

Is this shader publicly available. I'm new to the forum and I am wondering how to download and test? Thank you for your time/effort.

It's a hidden hyperlink in the first page, first post, letters HQAA at the start.
Or, yes, it would be better if it was an actual link, like  github.com/lordbean-git/HQAA , since forum does not highlight hyperlinks for some ungodly reason, so the only way to see one is to point a cursor at it.