(H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)

v18.6 now available, and I'd recommend updating even if you don't use the brightness gain optional feature. I implemented a Vibrance setting as part of the optional brightness gain effect, and discovered + fixed several problems with the way HQAA was calculating saturation adjustments in the process. The Hysteresis blending pass in 18.6 gives a cleaner and more accurate result because it also uses the saturation adjustment function.

Edit: to those wondering why HQAA has as many optional features as it does, this is basically an empirical summary of the reason. I'm not particularly skilled or experienced yet with ReShade FX and often it's the small optional additions that end up exposing a problem with something else I had done. I'm learning as I go

v18.8 now available (18.7 was a housekeeping update that got rid of a chunk of old code). 18.8 adds a new "Fudge Factor" feature to the hysteresis pass that allows you to configure the shader to skip hysteresis for pixels that didn't change much after anti-aliasing. Useful for running strong hysteresis correction in high-contrast areas while allowing low-contrast areas to blend their anti-aliasing result fully.

18.9 is now available. It's a generalized update aimed at longer-term usability and compatibility with other shaders, and includes a fairly major overhaul of the way the setup options (particularly the introduction) are presented. I have also revised all of the default settings (including which optionals are enabled by default) so that HQAA will produce good results even when deployed in a "fire and forget" fashion.

18.9.1 now available, which fixes a bug where the vibrance adjustment function could produce an out-of-bounds pixel in rare cases (would typically look like an oddly dark or bright patch in the middle of a bright or dark patch when it happened). The Hysteresis blending pass also calls the function, so it's worth the update as there should no longer be any possibility of an artifact being generated during the saturation phase of hysteresis adjustment.

Hey Lordbean,

Just wanted to say I really appreciate what you've made here, it's very impressive! I love how it performs and how clearly you've laid out the toggle-able options, which is a must for me (for e.g. I always use Custom with corner-rounding to 0, I try to minimise impact on text as much as possible, same goes for reducing 'crawling').

I can also sense HQAA is near completion and if you've enjoyed making it you may be looking for something else to cut your teeth on, and to that end I have two ideas for you!

1) Integrate pre-sharpen
In my testing over the years I've found I almost always get better AA results if running AA after a sharpen shader. I highlighted this a while back in a post about shader-ordering. While in theory a sharpen filter should run after to bring out anti-aliased edges, it's been borne out in testing that invariably running a sharpen shader first helps AA methods find and/or complete edges. Of course, with shaders like FXAA which tends to soften the whole image, it's also helpful to run a sharpen filter after as well which is what I often do. But back on topic:

While we can simply run a sharpen shader before HQAA, I wonder if there's some performant or quality advantage in integrating a sharpen pre-pass into HQAA -- i.e. passing on the shader mask to operate on directly instead of applying to the image first (I could be talking out of my butt here with how this works!).

2) Attempt a detail reconstruction
This is probably more out of my butt, but here goes: I noticed in some screenshots you posted a long time ago, and in my own playing with HQAA (and AA shaders in general) in games that distance detail can be lost -- e.g. think chain-link fences with criss-crossing lines at a distance, or ropes, or power lines etc anywhere where the resulting AA ends up removing data and they become incomplete lines, or dotted lines with inconsistent gaps etc. It's just a function of the process.

If my understanding is correct from the method of unsharp masking, this works by creating a mask, inversing it and applying it to bring out edges. If this were combined with a shader like embossing to create thicker edges, then potentially I imagine those said distance thin lines that erased by AA shaders could potentially provide more data to be restored to solid lines, thus preserving that detail. But wait! That would be terrible for the whole image, imagine a semi-embossed-strengthened edge all over everything! Indeed, which is why you'd use the depth buffer to apply it only at a distance, and temper/fade it out gradually as you get closer to the player. Realistically it'd only be functioning gradually somewhere from the mid-point to the the furthest distance from the player I imagine, slowly phasing in. I'm not sure if this is something you'd run after HQAA's main work is done to restore data, or before to help it smooth our re-generated lines, but working that out is all probably part of the fun!

Alternatively to an embossing technique to bring out the lines, perhaps using something like the unshark mask duplicated and offset a pixel in a direction (or two), creating a mask with thicker lines. Again, assuming my butt isn't sprouting gibberish here.

Of course, this type of thing is one reason DLSS exists, to restore detail while AAing, but maybe we can fudge it a little with something like the above methods
 

Thanks for the feedback Martigen, I appreciate the time you took to write the post. I believe you're correct, HQAA is likely approaching a "finalized" state at this point - the only major feature set left that I'd like to tackle before I call it "done" is HDR support. I am not currently certain whether HQAA works properly in the DisplayHDR color space (and this is one I know I can get it working with) since I've been in a situation where I'm unable to test it. Secondly, I'd like to get it working properly in the scRGB color space if possible - although this one is more difficult as it requires purpose-built conversion functions to go between linear and scRGB (which is also known as HDR10).

You might be surprised to learn that detail reconstruction is exactly what the Hysteresis pass does! Although it doesn't directly use a mask or the original scene, a key difference you can observe between HQAA and SMAA when you look at the statistics tab in ReShade is that the edges texture for HQAA is a full RGBA8 texture (SMAA uses an RG8 texture). HQAA piggybacks information about the saturation and luminance of the scene into the blue and alpha channels of the edges texture during the SMAA edge detection process, and then the Hysteresis pass (which runs after both anti-aliasing methods have completed) takes new measures for the saturation and luminance of the same scene, and then nudges the pixel toward its original appearance by adjusting the new saturation and luminance closer to the original values (the exact strength and tolerance of which is adjustable in the configuration).

Regarding using pre-sharpening, I would tend to agree with you that I typically notice better anti-aliasing results when using a sharpening pass first. The reason why I did not implement one in HQAA was (at the time I considered it, which was a long time ago in the scope of the project) because I did not want to do something to the scene that the end user might not necessarily want. However, since I've gained more programming experience, I believe this idea is very likely possible to implement - I should be able to configure a sharpening pre-pass to render to a texture, and then perform edge detection on the texture rather than the backbuffer. I'll experiment with it and see what I come up with.

v18.10 now available implementing above suggested pre-sharpening approach. HQAA now renders a sharpened copy of the backbuffer to the SMAA blending weights texture (avoids declaring a new one) then does edge detection using the sharpened copy instead of the back buffer (hysteresis data is still taken from the back buffer). This gives it the advantage of performing detection on a pre-sharpened scene while also discarding the sharpening so that you can apply your own sharpening as you like (via HQAA's optional CAS, or any other sharpening shader). It only has a minor performance impact since it uses a texture that was already declared but not used until after the edge detection phase.

Edit: now updated to 18.10.1, which is a largely debug-focused cleanup and polishing update, as well as some more minor performance optimizations and some upkeep towards longer-term code maintenance. This version includes a mode that shows you a pretty functional visual representation of what the Hysteresis pass is doing so that you can configure it more easily.

v19.0 now available. Changelog (copy-pasted from github commit):
While implementing a small helper function for edge detection, discovered and fixed a major problem that was causing SMAA to RADICALLY underperform. Although the new code in 19.0 amounts to relatively little, the boost in quality from the corrected SMAA is so significant that it warrants a full version rollup.

Version 19.1 update now available. HQAA now has full support for both HDR in nits and HDR10/scRGB modes (although for the moment it must still be manually enabled).

v19.2 now available. This is very likely to be a largely final release version as I believe all parts of HQAA now work 100% correctly, both primary and optional. Of significant note, the visual quality in v19.2 took another big step up over 19.1 due to discovery of a glitch that almost snuck past me.

v20.0 release now available. This version adds support for being used in addition to a game's internal Temporal Anti-Aliasing via the preprocessor flag HQAA_TAA_ASSIST_MODE. When enabled, HQAA generates and uses a luma histogram to detect motion in the scene, and runs only where motion is detected each frame. This allows it to cover for TAA's weak performance in high-motion situations while conserving GPU power by treating TAA results as acceptable when the scene is not changing.

Gotta say I'm excited to see you working on the TAA portion of your shader!

Sorry to bring it up again, but would you consider at some point looking into a depth-edge assist mode? Essentially restricting the effect to pixels in a radius around mesh edges.

I believe it would prevent the effect from smudging surfaces and distant views like night skies (TAA can really murder stuff like stars) and possibly save on performance depending on implementation.

A similar depth filter on debug view to illustrate (although the effect in question does the opposite and skips mesh edges):
imgsli.com/OTc1MDQ

A shader combining FX/SMAA+CAS for overall picture and a mesh-edge focused TAA would be my Holy Grail of AAs.

I might look into a possible optional use of the depth buffer at some point, yes. I've been focused on squeezing the limits of what is possible without depth data because I want a shader that can produce solid results in multiplayer games (although I'm not a competitive gamer, it wouldn't be unheard of for me to be doing some Borderlands 2 co-op or similar).

As an interesting observation regarding what I did in 20.0, I noticed an interesting positive side effect. When TAA isn't sure about something and causes it to shimmer, HQAA detects it as "motion" and runs anti-aliasing on it. This doesn't remove the TAA shimmer entirely, but it considerably dampens how much the TAA shimmer stands out.

v20.2 available now. 20.1 was a relatively minor update with some performance and quality improvements. 20.2 revives an old method I tried in the early editions of HQAA - multisampled FXAA. The corrections done by HQAA are at a sufficiently accurate level that FXAA multisampling is possible without particularly blurring the scene, and it is highly beneficial when correcting certain complex types of aliasing. I have also included a preprocessor definition that can be used to revert to one FXAA pass for systems where the performance hit from FXAA multisampling is too large.

Minor version 20.2.2 is now available which corrects a compile failure under DirectX 9. Oops.

What about cyberpunk 2077? The game without AA looks terrible. And it seems your shader is not able to compensate for this. I would look for an application which will require the help of antialiasing via reshade, but I'm afraid for mass effect or gta iv enough a banal SMAA. It's not that I don't respect your work, but in my opinion, absolutely any shader should be useful and a priori be used, while looking at many games, I don't find any use for your shader absolutely

I don't think I follow your train of thought. Any game that includes a Temporal AA solution typically isn't fixable properly by a non-temporal routine because TAA does a significant amount of denoising correction for other effects.

v20.3 now available. This is mostly an HDR support update - I fixed a couple of problems in the PQ transfer functions used for HDR10 + scRGB and implemented an additional PQ approximate mode that does the transfer much faster but less accurately than the normal method. All PQ transfer functions now assume a maximum encoded luminance of 500 nits if the buffer is in RGB10A2_UNORM and 10,000 nits if the buffer is in RGBA16_FLOAT as this seems to produce reliable output in Far Cry 5 on my monitor (I'm still somewhat unsure what the encoded ceiling is supposed to be in RGB10A2_UNORM though. From my research I get the feeling that this mode is mostly used by low-end DisplayHDR400 monitors, but that impression may not necessarily be correct. If you experience problems please let me know).

20.4 now available. I made some improvements to the accuracy of SMAA edge detection and have adjusted the default settings around them. Also fixed a compile failure when attempting to enable screenshot mode.

v21.0 now available. Implemented a new error reduction feature (after the initial detection pass, the detected edges have isolated positives removed to maintain detail from the original image), did a major rework of the preprocessor definitions setup towards the overall goal of keeping things simple, and got all debug outputs working correctly when in TAA assist mode.