(H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)

dxgi.log

Here's my reshade log (I assume this is the correct file). Let me know if there's anything else I can do to help identify the issue.

Edit: the issue is less pronounced in the newest version, and looks slightly different from before, though still similar in the actual edges - imgur.com/a/Bp1fYSx
 

That is the correct file, yes. Interestingly it appears to be running in standard RGBA8 mode. I'll check and see if mine does the same thing.

Edit: It does. As far as I can tell the game's rendering environment is the same on both our systems, apart from the GPU being different. On that note, are your drivers reasonably up to date for the GTX 970? I wonder if I've made my shader do something that's throwing an older driver off. Also, do any of the debug modes isolate the artifacts on the screen? I strongly suspect the problem is coming from one of the FXAA passes at this point.

I'll check in a bit and update this post. Drivers should be just one behind the latest, but I'll update just in case.

I've put another compatibility option in for you to try also, in v10.5.1. You can now set HQAA_USE_SPLIT_FXAA_LOOPS to 0 to go back to the old style of a single loop per FXAA per pixel to see if it fixes the problem. I think that was the biggest change in v10 compared to v9 - I broke the FXAA loop into two loops (one for each direction, positive and negative along the edge) which resulted in a significant performance improvement. It's possible that the dual loop setup isn't playing nice with some GPUs though.

All debug modes seem to align, the artifacts are the detected edges, basically (from my understanding).

Messed around with all preprocessor definitions at 0 and 1 (including the new one), but sadly the same still happens.

Alright, one last thing to try for now, if this doesn't fix it I'll have to let it stew for a while as I'm all out of ideas now. v10.5.2 reverts a small change I made in the way FXAA does texture accesses which was also new in the version 10 update.

No results on my end, sadly. Tried a clean reshade reinstall just in case too. Do let me know whenever there's anything else I can try to help. I'll try to find the latest version that works for me and inform you too, and settle with that for now with this game.

Update: 9.7.2 works fine, and 10.0 is indeed the one that breaks it on Witcher 3.

Man that's frustrating. Ironically though I think the original way the routine did its texture accesses produces a more stable result than what I did in v10, so I'll probably just leave it alone going forward either way.

Settled at 9.7.2 for Witcher 3. since it's indeed 10.0 that starts showing issues in that game.

This AA does wonders for the game at 1080p, it's really excellent, undoubtedly the best available AA for Reshade.

Comparison (check in fullscreen of course): HQAA OFF vs. HQAA ON + Strong Adaptive Sharpening + Slight Gaussian Blur

Still tuning my sharpening, I like it a bit on the stronger side for this game, but it's a bit much right now.
If you see any edges that were touched weirdly, it's because AA applied on top of the game's weak temporal AA sometimes causes that, though I found that your shader in specific tends to do that much less frequently than the individual SMAA/FXAA shaders, which is again, excellent. The game's TAA is basically necessary, otherwise any walk through the woods is a shimmering nightmare that will leave a mark in your mind. It's weak enough that it has no movement blurriness, though does smear the screen a little bit but removes most egregious shimmering from foliage.

Thanks for you work in this, and do take your time like it has been suggested. When some days passed and I suddenly came back to a 10+ version after using one of the first versions for a while, I was quite impressed.

Haha, the updates to HQAA are probably going to start coming slower in the future just because I think I'm running out of ways to coax better results out of SMAA and FXAA without actually starting to delve into how their actual anti-aliasing math works. So far everything I've done has been one of either two things - help them to detect edges more accurately, or make them smarter about how they blend their final result. I haven't yet made any attempt to learn how the actual AA math works (though I have a fairly good empirical understanding of what FXAA does at this point, I still haven't actually broken apart the specifics of it).

Lord of the Bean delivers and hopefully keeps delivering. Although I'm stuffed with chicken. No more that, please.

The updated debug options are very nice but I'm not sure I understand the difference between MSAA negative and positive options. Care to elaborate on it a bit?

The additional CAS shader is pretty much useless on 1080p on default range of values(assuming it is resolution dependant). Cannot see much of anything if not cranking it up to 11. But it's not really even needed. The one sharpening pass in the middle produces great results with moderate values. Scratch that. The problem seems to be a game specific. For example Green Hell seems to get very strong sharpening while in Watch Dogs Legion the effect was very miniscule. Might be because the former is much more dense with small details and there seems to be a noise shader on top of it.

Basically, the two FXAA passes are using the edges detected by SMAA to determine where to run. SMAA positives runs first, and it will only run edge correction where there's an edge recorded in the detected edges texture. The idea is to do a sort of final smoothing-out since SMAA blending sometimes leaves a fairly strong level of hinting after completion, and I've found FXAA is good at hiding it. Finally, the SMAA negatives FXAA pass only attempts to run correction where SMAA didn't record any edges - it essentially is a final sanity check because the way it calculates its result blending, it will tend to dampen errors (correction applied where there wasn't really an edge) and make them look more like the original scene. That's why when you click between them on the debug options, they look a bit like inverted versions of each other.

HQAA v11.0 now available on my github page. FXAA had one last dirty trick up its sleeve to teach me when I went reading through the FXAA source files. HQAA v11.0 performs one additional pass (it's the new first pass in the shader) that calculates normalized luma values for the scene and stores them in the alpha channel. Since I had already integrated the alpha channel heavily into the edge detection processes in the shader, this had the effect of radically improving HQAA's ability to discern real edges from background noise. Since it's much more sensitive, HQAA v11.0 is somewhat slower than prior versions, but it produces extremely accurate results.

I'm interested to know whether the problem with Witcher 3 still exists after this update, also. I can confirm HQAA v11.0 works on the Witcher 3 on my own PC, but so did the v10 series, so this doesn't necessarily mean much.

Edit: standing version is now v11.4, which has a few new tricks on a smaller scale. HQAA v11.4 is modestly slower than v10 versions but not much slower (at least while using a preset). HQAA is now in a place, however, where pushing the shader harder actually results in increasing levels of AA accuracy without introducing artifacts (everywhere I've tested it so far), but the trade-off is that when using custom to really push for max quality, HQAA can be a very slow shader.

Hi. Tried latest version 11.4
and it's look like this in dx10/11 games

i tried dx9 game, and there aa just don't work without artifacts
also at first hqaacas shader was marked by yellow, some error, i wasn't able to screenshot it, wanted to just make photo, but after restarting game error not recurring, maybe it related to some shader cache.

Also reporting an issue with 11.x series -- the 'Ultra' and 'Glados' presets cause text to 'crawl', it looks like pixels are rapidly moving around the edges. This doesn't occur with 'High' and below. Testing, it's the Threshold value when it gets less than around 0.060, with the artifacting worse the closer to 0 you get.

This didn't occur in the 9.x series (and maybe 10.x, I'd have to re-check).

It actually did occur rarely in any prior version, Martigen. SMAA tends to do that when it's running at a high sensitivity, which HQAA does its best to stabilize by using FXAA interpolation blending (primarily in the negatives pass) as a result sanity check.

Regarding the reports of artifacts, I've rebuilt the new alpha channel shenanigans to use a support texture instead of the back buffer alpha channel since it's possible that was the cause. v11.5 should hopefully resolve that problem for anyone who's been getting artifacts since 11.0.

I've checked 11.6.1 and it still artifacts (look's like nan pixels when i debugged offline rendering system)
in dx9 it's actually happens too, it just not black

also the error i mentioned before, that won't reappear after restart of game

warning X3571: pow(f, e) will not work for negative f, use abs(f) or conditionally handle negative values if you expect them



 

Thanks very much for calling that out, for some reason my own system had not managed to show me that warning. Your suggestion of NaN pixels could be on to something also as I occasionally used rcp calls during calculations - it's possible that some hardware configurations might be getting thrown by a NaN ending up in the results (a possibility I hadn't even thought of until you mentioned it just now). I pushed a small update that replaces all rcp calls with a conditional rcp function (if an input is zero, it won't run rcp on it, it'll just return the zero).

checked out 11.6.2 but sadly same artifacts
no error tho, at least after rewriting file in already runs game
-
may be i should provide some technical info to help you debug it?
i guess only related hardware configuration is videocard - i have good old gtx 1070, and windows 11
i remember when my friends wrote render system, there was problem when same shaders was ok on GPU but was buggy on my, and they needed some manual fixing, as i understand.

Hmm. I wonder if some GPUs don't like doing per-component interpolation?

I'm gonna slap a quick and dirty compatibility flag in for you to try. It'll be up shortly now. v11.6.3 implements a pre-processor define allowing to disable per-component interpolation (used in both sharpening functions) and fall back to traditional single-float pivot.