(H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)

v7.4 is now up, it fixes the logic flow used in the SMAA neighborhood blending calculation for the result sharpening and now compiles on DirectX 9.

That was quick, well done!
Your shader is great btw!

v8.4 now available. Significant changes since the last revision I posted here include a new method for SMAA luma edge detection to calculate its color channel weights and determine whether to run or fall back to color detection (as of 8.0) and significant performance optimizations (as of 8.4) gained from rewriting many of the math operations in FXAA (and a couple in CAS) to take advantage of hardware-accelerated results when possible.

HQAA version 9.0 now available.

SMAA edge detection (whether color or luma) has problems with scenes that are very blue or very unsaturated (grayscale). To work around this, HQAA now runs luma edge detection a total of 4 times to construct its edge data.
First pass: uses normal backbuffer as input
Second pass: uses a copy of the backbuffer with the color channels left-shifted
Third pass: uses a copy of the backbuffer with the color channels right-shifted
Fourth pass: uses a copy of the backbuffer with the negative of the color channels

Due to these changes luma edge detection will no longer attempt to fall back to color edge detection, it will simply return no edges if it does not detect any. Since this removes a fair amount of math from each iteration of luma edge detection, performance of this version is not significantly worse than version 8.

Since each new source of backbuffer data requires a copy of the back buffer to be written, the HQAA technique's expected number of passes is now 11 when inspected in the statistics tab.

I haven't done a showcase montage in a while, so here's a few before/afters (click-through to visit imgsli.com for the comparisons) taken using HQAA + HQAACAS version 9.2, all default settings (High preset / 1.0 sharpening). Images are 2560x1440 (aka 1440p aka 2K aka QHD).

Sorry, I come again complaining.

There is something funky happening with the green colour. The effects can be seen in the Rage2 near the green neon sign. Specifically in the rain cover or whatever that metal plate is called. Another one is on the truck game on right where the street sign shafts are against the green backdrop.

Most of the time there seems to be no problem with green but those small anomalies instantly caught my eye. Maybe something with bluish grey and green together with high luma change in between?

On the other hand, on my fullHD screen I have nothing to complain about. Haven't seen such clear anomalies. Also, it has been effectively impossible to see things getting better after about versions in page 3 but I think that's more a function of low resolution than anything else.

One request, though. Could you enable debug views for different textures to help making more informed decisions and trouble shooting?

There's actually a good reason in this case the images I posted have some obvious missed positives - I hadn't gotten the local contrast adaptation quite right yet after revamping edge detection to be performed on multiple different presentations of the same scene. The most recent version (9.2.2) actually calculates it dynamically per-pixel, which I haven't tried before up until now - but it seems to be working quite well.

As far as troubleshooting goes the aspect of HQAA that currently has no effective way to judge other than purely visually by the result is what the FXAA passes are doing - since there are no steps in between detection and blending for FXAA (hence why it's fast), there's nothing to output in the middle to see what it's actually doing. I might try fiddling around with some sort of negative output mode though where instead of rendering the correct result, the debug mode will output the negative of the result (which will be glaringly obvious on the screen).

For SMAA, seeing what it's doing is still actually pretty straightforward as long as you know how to read the data in the edgesTex texture on the statistics tab (the one that's in RG8 format at the same resolution as the back buffer). Green means edge detection has flagged a probable horizontal edge and red means edge detection has flagged a probable vertical edge (yellow means it's flagged both, which will likely be picked up by the diagonal processing functions). The changes I have made to the core workings of SMAA are still purely in the first step of detecting where the probable edges are (with the one notable exception of implementing sharpening of the final result that blends with the screen at the end of the process). Everything that SMAA is doing inside of HQAA after it completes the edge detection process is otherwise still totally by-the-book SMAA.

Edit: of minor note also (as I imagine you've likely noticed it, but just in case you haven't), imgsli.com uses linear scaling to resize the source images to fit the window, so they can often look weirdly distorted when it's shrinking them (which it often is, unless your monitor is higher res than the screenshots). You can target a zoom-in by rolling the mousewheel up with the pointer moved to the approximate area you want to see, which will give you a much closer inspection of the images.

Hi. HQAACAS shader from latest HQAA.fx still doesn't work for me (last game I've checked is The Evil Within 1, its dx10/11).
CAS slider is working, but no actual AA applied, only sharpen.

That's what the HQAACAS part does. It's an optional sharpen that runs very similarly to CAS.fx with the bonus of being aware of where HQAA ran anti-aliasing. For the anti-aliasing part, enable the HQAA technique.

Tojkar wrote:

One request, though. Could you enable debug views for different textures to help making more informed decisions and trouble shooting?
 

Done. v9.3 is up and includes SMAA debug modes identical to SMAA.fx as well as experimental (but honestly hard to read for the moment) FXAA debug mode.

Please stop apologizing for complaining, your complaints have typically been very helpful to me.

v9.3.1 is now up. I split the FXAA debug option into 3 sub-options (one for each FXAA pass) and discovered that both the spurious pixel pass and the SMAA negatives pass were being a bit over-aggressive with their blending, and have thus reined in their enthusiasm a bit.

Edit: I just found a way to make the FXAA debug outputs readable, too - now available as v9.3.2.

Can we get another chicken dinner for Tojkar?

Got it pretty much polished and feature complete in v9.4.2 unless I make some other breakthrough in detecting edges more clearly. Quick example below taken using Custom settings at an absolutely bonkers level sensitivity and correction, and it still leaves a usable image after it finishes as well as catches flipping near all the jaggies I can see in the before shot.

Hello. How can I try this shader?

Look at the 1st row where the thread starts.

I think it should be a bit more visible for people.

v9.5 is up, which is pretty much a polished and completed version of all the work I've done so far. The changes are mostly to the UI - clarified descriptions where I could, sorted things more logically, etc. I also added two more input-variant edge detection passes to cover as many different color patterns as possible without having missed edges (they run surprisingly fast, though I kept the code to generate them very simple for just that purpose).

As klotim mentions, there's a standing link to it in my first post. I actually can't edit it due to this board's automatic edit time-out after 12 hours or so, and I kinda wish I could because some of the information that was originally correct is out of date (you do not need the FXAA and SMAA headers anymore, only the SMAA search textures).

Edit: we're now at 9.6.1, which is worth the update from 9.5 or below. I came up with a new trick in FXAA that eliminates the bulk of the overcorrection errors it used to produce, resulting in an even cleaner final image than before. I also allowed the custom preset options to go way beyond what the internal presets do for those who want to see older games look their absolute best on their $2G+ GPU (this comment is only mildly sarcastic, I bought a 6700XT for 30% more than MSRP. In most games at 1440p, I have a ton of room for GPU-powered post processing).

@klotin LOL Thank you very much xD It was the # HQAA

@lorbean I will try this shader, thank you very much for your work.

Version 9.7 beta now available, I need help testing this one. I put in experimental support for HDR displays but I don't own one. I'd appreciate it if someone could test whether it works or not and let me know.

Edit: now also includes potentially significant improvements to the FXAA code. It now does a full neighborhood luma check to decide whether to early-exit or not (instead of just a + pattern) and I converted a lot of the per-component operations to vector math.

Amd has a solution for HDR CAS, here's a HDR CAS for reshade, got it some time ago:
codeshare.io/8pn3pv

Its available 24h

Question, how does the AA behave with different aliasing scales, like a large aliasing scale compared to a small aliasing scale? I guess large scale anti-aliasing could cause false detects?

ok, i get it now, i just thought it's just alternative aa+sharpen shader
this mean i should set same settings for AA as it set in HQAA shader?
nvm, it have same settings for both shaders in latest version, i think it was separate before

lordbean wrote:

Version 9.7 beta now available, I need help testing this one. I put in experimental support for HDR displays but I don't own one. I'd appreciate it if someone could test whether it works or not and let me know.

Edit: now also includes potentially significant improvements to the FXAA code. It now does a full neighborhood luma check to decide whether to early-exit or not (instead of just a + pattern) and I converted a lot of the per-component operations to vector math.
 

Still does not work in HDR, the preprocessor definitions don't seem to do anything. HQAA (9.7) minus the sharpening pass kills the highlight brightness, with the sharpening pass the highlights are inverted.

Here is what it looks like: (tested on an LG CX)

imgsli.com/ODk1MTY