(H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)

Oof, that's clearly wrong. I'll see if I can correct that.

The preprocessor definitions are supposed to disable SRGB sampling and rendering when HDR is active. Could be that some of the math the passes are doing doesn't work well with HDR data though.

Pushed an update to 9.7 beta which should theoretically fix HDR for at least the CAS passes. If I'm lucky the AA passes will just work, if not I'll investigate what I need to do to un-break them for HDR. Thank you for the screenshot Riadon, it's very helpful to see those in this process as I can make an educated guess at what pieces of code are generating bad results.

Edit: my ViewSonic monitor was able to produce garbage results in Far Cry 5 using the HDR setting "FreeSync 2" using the prior version. After the update, and having set HDR_BACKBUFFER_IS_LINEAR to 1, it renders properly. Hopefully this means I fixed it.

Also pushed another update with an improved FXAA interpolation calculation and a new debug option to show FXAA luma color channels (basically, it represents which color FXAA selected as being representative of the brightness of the pixel by my adaptive luma code).

That update fixed it, working perfectly in HDR now.

Very impressive for being done purely in post-process. Doesn't really touch temporal aliasing but in every other aspect it's miles ahead of any ReShade AA shader I've previously tried. Great work!

Riadon wrote:

That update fixed it, working perfectly in HDR now.

Very impressive for being done purely in post-process. Doesn't really touch temporal aliasing but in every other aspect it's miles ahead of any ReShade AA shader I've previously tried. Great work!
 

It likely never will touch TAA either, at least not when the TAA is implemented well. It isn't really intended to compete directly with TAA though - my reason for working on HQAA is because Temporal AA starts migraines for me (it's a physiological reaction like motion sickness from motion blur), but for most people the target use case is games where TAA isn't available.

9.7 release now up. I made a further improvement to the FXAA interpolation calculation which allowed room to remove the clamps on the subpixel correction strength passed to the FXAA routines. FXAA luma debug also improved to show approximate brightness of the detected luma (lighter color means brighter luma and vice versa). I also found and implemented another performance optimization in FXAA, although it was a relatively minor one.

v10.1 now available. I skipped posting version 10 because although it had a whole lot of changes, they were mostly small in terms of the actual output visuals. 10.1, on the other hand, has significant improvements to both the SMAA edge detection math and the FXAA blending calculation math. Even on the highest possible settings, HQAA v10.1 leaves text and fine details virtually identical to how they started.

Edit: I discovered a previously unknown (to me, at least) relationship between the FXAA scanning granularity and subpixel correction strength, which has allowed me to develop a more flexible way to calculate it to adjust dynamically with the granularity setting. Additionally I managed to improve the performance of FXAA a bit more by splitting its correction loop into a branched loop structure instead of having a single loop handling two different jobs. This is available now (version 10.3).

2nd Edit: This is probably pretty close to a final version of HQAA now. I literally maxed it, dropping my fps from 165 to 93, and I see extremely few (if any?) AA-produced artifacts in the output image here.

well done!
Can the search/area textures be improved maybe?
Also in your image, left sidewindow you can see the reflection on the truck, if you zoom in there you can see the stepping of the aliasing in after as well, is it possible to work to make it a straight line?

The short answer is that the correction is never going to be perfect because the code doesn't really "see" lines, it just sees numbers and calculations to make with them. v10.3 and up does tend to leave just a tiny bit more hinting where it corrected aliasing, but that's due to the same change that made it leave text and details almost completely alone - I'm basically crunching out a "normalized" luma for the same pixel in 3 different states in the final part of FXAA (unmodified, post-SMAA, and post-FXAA) and then determining how to blend the FXAA result using those data points.

Edit: I might eventually try to tackle improving the textures (I'd like in particular to see SMAA able to scan further than 20 steps for diagonals) but I frankly don't understand SMAA well enough to try it yet. The techniques used in SMAA are much more complicated (like orders of magnitude) than what FXAA does.

Try this on for size though - this is taken using shader defaults, v10.3.3 (High preset +1.5 sharpening).

Any idea what could be causing it to look like this? Default settings used, and I did download the textures to the correct path.

imgur.com/a/VDlOGsI

If you're using HDR, you probably just need to set the preprocessor definition for it to 1. If that doesn't fix it let me know and I'll investigate further.

Not using HDR, though tried that anyway and it didn't work, sadly. I'll check another game just in case when I can.

Edit: Tried another game, seems to be a Witcher 3 issue with it, which is sad since that game really benefits from this shader (used previous versions on it). Not sure what exactly causes it but I'll mess with some settings in the hopes of solving it.

If you wouldn't mind too much, could you check what the back buffer format is for Witcher 3? I suspect I know what's wrong and it has to do with a sort of faux-HDR color-space called scRGB. I've discovered if I disable FreeSync in my monitor, Far Cry 5 shows me scRGB as an HDR option, and my shader doesn't work with it, pre-processor set to 1 or not. With HDR modes HDR10 or FreeSync2, my shader works.

If Witcher 3 is using scRGB color space the backbuffer should be in RGBA16 mode. I'm fiddling around with trying to convert, but it's proving tricky because scRGB uses the same fused conversion as sRGB in hardware, but its range extends 0..1 significantly (it's typically -0.5..7.4999999).

Edit: The problem is coming from the contrast adaptive sharpening functions (they rely heavily on reciprocals of fractions). For the moment, I've pushed version 10.3.4 which disables all sharpening code in the shader if the back buffer format is RGBA16. If it fails to compile, you just need to delete the saved pre-processor definition from the ReShade profile ini file. You'll need to use sharpening from another shader, but it should compile on Witcher 3 now if my assumptions are correct.

Is that something I can check on reshade itself?

Edit: Saw your edit, gonna give it a try and edit this to update it.

Update: Unfortunately still looks the same. If it's of any help, the separate sharpening shader part works just fine but I think it did before too. When trying the main shader I made sure to disable the sharpening manually just in case too. Hopefully someone else here has the game available to try it, maybe it's something on my end, though it did work on other games just fine.

Dang, I was hoping that was going to be it. I haven't bumped into another case where it doesn't work yet. The fix I put in got it working correctly on Far Cry 5's scRGB mode, but there must be something else going wrong when it loads in Witcher 3.

v10.4 now available, I put in several more debug modes to try and isolate problems in specific games. The most important sanity check is the original buffer copy, if it doesn't look identical to what the game shows onscreen with ReShade effects disabled something is going wrong. If anyone bumps into a case of this happening, I'd love to hear about it - ReShade log file and/or screenshots a big plus.

Edit: and of course anywhere else it doesn't seem to be working.

I just discovered I apparently own Witcher 3. I thought I had refunded it (couldn't stand the locked 60 degree FOV) but apparently I didn't. I'll queue it to install and mess around with it tomorrow to see if I can pin down the problem there.

A request if you don't mind.
I would like to be able to disable sharpen with a preprocessor checkbox because of 2 reasons:

1. It would be nice to disable/enable sharpen with preprocessor toggle, as i understand it saves some performance.
2. Disable sharpen in preproc would also brute disable sharpen no matter what preset I use.

Reasonable enough, now available as 10.4.1. Note that the pre-processor define will only disable the code used by the inline sharpening for results, you can just uncheck HQAACAS to turn off the other part.

v10.5 now available. Sorted the UI so that it presents visually a bit better and improved the FXAA blend calculations.

Seemingly unable to duplicate a problem with Witcher 3. Maybe I fixed it by accident between last night and now?

I have the same issue as Kidoki, but for me it started from version 10. The 9 was fine.

Anyone bumping into a problem with The Witcher 3, it would be really helpful to see the ReShade log file from the game. I need to determine if the game is choosing different output modes on different PCs through some automatic process because it seems to work fine on my own PC.