(H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)

The new softening is absolutely disgusting. Make it a choice between algorithms if possible.

The ever-atrocious one pixel dot problem is still omnipresent despite softening, in fact, it makes it even worse since since everything else gets smoother but dots are still obvious.
Examples:
imgur.com/b109sX9 - dots in the trees, both left and right, HQAA applied, no softening
imgur.com/FLuE36h - softening on, dots are still there, just SLIGHTLY less pronounced
imgur.com/jTGy1yx - this is pronounced even in less vibrant games, such as FEAR, for example.

How I fix this BS in games with robust and highly pronounced details (read - cell shading titles, e.g. Firewatch or Borderlands, which do not suffer much if you blur the heck outa them):
imgur.com/HmMQ0Bt - add FXAA with maximum (1.0) subpix and 0.0 Edge Thresholds; it has serous drawbacks though, standart FXAA shenanigans
imgur.com/YbEftL7 - apply HQAA with softening in top to smooth the result; the BLUR, though, is overwhelming, therefore, as I mentioned, it is only viable for games with robust, pronounced details, of which cell-shaded or cartoonish titles are the most typical representatives since everything significant is outlined. But this method is only viable for a handful of games which can be counted on one's fingers. This needs a better solution viable everywhere. I know I already mentioned this problem before at least twice, but it is very annoying and very in-your-face in quite a huge number of titles, since most games have, in some shape or another, foliage such as trees, which have leaves, and grass, which produce this defect en masse.

The softening effect is still a fairly work-in-progress sort of thing - I was honestly surprised the approach worked well enough to keep in the first place (I think I mentioned that in the version update post iirc).

Btw, you can download any previous version you want from Github by looking at the commit history for the file. You have to jump through a couple of extra hoops to get it to show you the old file as it was without doing the change highlighting thing, but older versions are all still there.

Thanks, I have a few earlier versions, including 27.5 stored in a folder due to my habit of not getting completely rid of older versions unless the new one is absolutely superior.

The new softening looks very much like DLAA on textures, and not in a positive way. Older softening was closer to gaussian filter you get from nVidia's DSR when setting it to 50% or above smoothness, in a good way. It would be best to just put these under the drop-down choice menu, similar to that of HQAA quality level. Or maybe make it so inputting "1" into the field enables new softening mode, and "2" switches it to the older version.

Any thoughts about implementing your version of CMAA?
github.com/LordOfLunacy/Insane-Shaders/b...er/Shaders/CMAA_2.fx

May look into it eventually, but compute shaders are outside of my scope of understanding currently. Still pretty new to shader coding.

I just pushed 27.5.5 with what are hopefully improvements to the image softener. Realized I had an incomplete set of geometric patterns being used for horizontal/vertical, which might have been negatively affecting the output.

27.5.6 now up which adds some basic tone mappers to the brightness & vibrance optional feature. Nothing too fancy, just fast ones to apply.

v27.5.7 now available. Image softener sample patterns can now be expanded or contracted around the central pixel using a new offset value in the menu. This should help fine-tune it for good output in different situations.

v27.6 update now available (lite version included). When using advanced mode setup, FXAA can now be forced to process the entire scene. This shouldn't be necessary in most cases, but might help in some rare scenarios. The optional image softener and temporal stabilizer have also seen further improvements since my last post here.

v27.7 update now available. This update is relatively minor, but is worth doing because I found and corrected a bug in the edge error reduction pass. There are a few other changes as well, mostly along UX and final polish lines.

Current version is now 27.7.3, which has improvements to several of the optional effects HQAA offers. Of most note, number of debanding passes can now be configured (used to be hardcoded to two, now accepts one to four) and I've made considerable improvements to the contrast washout correction calculations when using the brightness booster.

Hi, how can i use this feature? i dont find it in reshade

It isn't part of the default ReShade packages. You can get it here:
github.com/lordbean-git/HQAA

Today I found another use case for an early (27.5) release of HQAA, specifically it's Optional Softening function.

Not sure if it was a bug or a feature, but when you select an "HQAA_TAA_ASSIST_MODE" option in 27.5, it disables the normal antialiasing function, but the Softening is still enabled (can't do that in latest versions). Not a big deal on it's own, but you can slap any other AA shader on top of that softening, obviously. So, armed with "HqaaImageSoftenStrength" = 0.750 and "HqaaImageSoftenThreshold" = 0, I went on and SMAA'd it on top, using minimum "EdgeDetectionThreshold" (0.05) for it to better grab a hold of those thicc edges. The quality was there, dare I say, at least not too far worse than the original HQAA with softening, arguably even better(?), but(t), so was the blur.

Just then, a brilliant idea shined upon me. As people here know, for quite some time already there's a built-in supersampling function in videodrivers (DSR for nVidia, VSR for AMD). So why not use it? Well, for me, it is quite limited. Unfortunately in the January of 2021, due to severe deficit on the GPU market (#f*ckyouminers), I was only able to afford an RX 6900XT ("only" because it was, like, 40% the price of 3090 where I live, and I damn was not going to waste half a kidney's worth of money for a laughable 10Gb of 3080). And the maximum AMD goes in that department is 5K (5120x2880), which is not even an 1.5x increment from my native 3840x2160. To add insult to injury, nVidia has a configurable built-in softening function with their DSR (DSR Smoothness, as far as I remember from my experience), while AMD does sad and soggy default bilinear interpolation, you can't even set it to Lanczos like GeDoSaTo. So the function just sat there pitifully disabled.

But now, with the newfound strength of Soft HQ-SMAA hybrid, I turned it On, and that is so good! Yes, you can still see the blur if you look for it, after all it's not even a 1.5x supersampling factor, more like 1.3(3), but it's still REALLY good.

PROS: amazing quality, the only things better are probably straight on 8x+ SSAA, 2.5x or higher factor DSR with 33~50% smoothness and maybe DLSS(?).
CONS: the performance hit is palpable, F.E.A.R. drops down 100+ FPS sometimes (although using just HQAA at 5K produced similar results, so it's an inherent demerit).

(P.S.: Yes, while downscaling, the difference between using SMAA, FXAA, HQAA or the loathsome soft hybrid is relatively miniscule, but this "relatively" is what can make your experience. Even at 5K->4K downsampling the imperfections of FXAA are visible, you can see SMAA with too low edge detection threshold flicker on edges, HQAA is mostly fine (e.g. Postal 2) but depending on the game may not be the better option (e.g. F.E.A.R.). When I still ran a GTX1080, I used DSR proactively in very old games that can still get 60+FPS with it, and these visual defects start blending in and become actually unnoticeable only at very high supersampling factors, along the lines of 8K->4K downsampling, that's why I'm so excited about the combined use of HQAA Softening with SMAA on top).

Sorry for the wall of text, I just fail to explain such things coherently in a few sentences. You can actually put any AA shader on top op HQAA Softening, just position it after the HQAA in the list. But SMAA with very low detection threshold seems to be the most bestest option, arguably better than even a second iteration of (non-softened) HQAA or HQAALite. It seems like because all the edges and details were thiccened by Softening, SMAA no longer needs to frantically recalculate positioning of (sub)pixels, eliminating the jittering which tends to happen on some edges and sharp details in cases of too low edge detection threshold.

TAA Assist mode doesn't disable HQAA anti-aliasing, it causes it to ignore pixels that had zero (or near-zero) luma change since the last frame. This is because TAA is exceptionally good at still image processing, but tends to leave jaggies on moving objects, so I built TAA assist mode to apply anti-aliasing only where there's probable movement in the scene. This is still the way it works as of now.

In other news, version 28.0 LTS (long-term support) is now available. I'm more or less considering HQAA to be done at this point, and future updates are only likely if I discover bugs or optimizations. While preparing the code for this release I was able to shrink the total number of compiled instructions of the SMAA edge detection function considerably, so v28.0 should actually be slightly faster than the v27 strain as a bonus.

Can you maybe extract the 27.5 and earlier versions Softening into a separate shader? It would be really great to have it as a separate function. 

Fair enough... wish granted.
github.com/lordbean-git/reshade-shaders/...aders/ImageSoften.fx
This is ripped from the original version in HQAA v27.5 since it didn't use any SMAA data. I added an extra pattern to it, removed the silly contrast threshold thing I tried to do, and replaced that with the ability to scale the sampling offset that I added in later versions. You'll have to recalibrate it as compared to whatever you were using in HQAA, but it should be reasonably flexible.

Thanks a lot!
I didn't even need to change the values. This seems to look exactly like 27.5's SoftenStrength=0.75, SoftenThreshold=0 with the default settings of 0.75, 0.75, or at least so close to it that I couldn't really tell the difference. The performance impact is also significantly lower compared to HQAA's TAA_ASSIST mode.

Here are a few screenshot fragments for comparison:  imgur.com/a/Ohfq6IU  
(FEAR 2 has 7680x4320->3840x2160 lanczos downscaling via GeDoSaTo, FEAR has 5120x2880->3840x2160 VSR + Softening + SMAA)

28.2 LTS update is now available.

I discovered an improved way to calculate the local contrast adaptation factor during edge detection, resulting in a significant increase in the function's sensitivity to edges. If you are using advanced mode setup, you will need to use the debug outputs to recalibrate your options after updating to this version as your old settings will likely cause a considerable amount of overdetection.

Additionally, I have also made the brightness booster optional feature available as a separate shader since I've come to use it a lot of the time. This is for cases where for example you are using DLSS or DLAA and therefore do not need HQAA to fix any remaining jaggies (QXAA or even a plain old FXAA will typically work for this as DLAA is very accurate in most cases). It's a fairly fast shader, requiring slightly less time to compute than a normal AMD CAS pass.

28.3 LTS update now available. It includes a lengthy list of visual quality improvements I'm not going to list here because the full details are available on the Github commit. In a nutshell, it has improved detection of low luma edges, better handling of spurious bright pixels, and preserves detail through to the final result better.