(H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)

  • anontsuki
More
2 years 4 months ago #21 by anontsuki Replied by anontsuki on topic (H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)
Read through the posts and updates you have made and inspected the comparisons, the shader looks great, and I have plans to try it out soon to compare and check out how intense it is.

I'd like to ask however, if you have any intention of making it focus on a depth buffer instead of full screen anti-aliasing? FXAA is great (crazed opinion, I know), but due to its effects on text, kinda sucks to use in text heavy games.

The small samples of lettering in your examples shows a minor modification to text, so it may not be bad, but would love to personally a depth only uh... shader, I guess.

Thanks either way, it's always nice to see new and different takes on aliasing with what we have.
The following user(s) said Thank You: lordbean

Please Log in or Create an account to join the conversation.

  • lordbean
  • Topic Author
More
2 years 4 months ago #22 by lordbean Replied by lordbean on topic (H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)
I prefer to avoid use of depth buffer information since I sometimes play co-op multiplayer games. As for text, you're definitely going to see changes in how it looks using HQAA, but I've tried to balance it well enough that you should always be able to read it even after applying the shader. I hate blurry text as much as the next guy.

Please Log in or Create an account to join the conversation.

  • lordbean
  • Topic Author
More
2 years 4 months ago - 2 years 4 months ago #23 by lordbean Replied by lordbean on topic (H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)
HQAA version 2.0 2.1 has been released and is available on my github now. It incorporates the header files for both FXAA and SMAA inside its own code to allow for easier future modification, and is now guaranteed 100% compatible being loaded in the same list as traditional SMAA, FXAA, or my other variant shaders.

Significant changes:
- Incorporates adaptive luma color selection in FXAA (defaults to green, but can change to blue or red if it has no green signal)
- Incorporates an in-line sharpen in FXAA before result is returned (this allows HQAA to sharpen only what FXAA runs corrections on)
- Implements use of ReShade standard discard call when FXAA enters its early-exit codepath
- A few other minor optimizations giving HQAAv2.0 slightly better performance than HQAAv1.x
-v2.1: now performs SMAA edge detection using only the Luma detect pass instead of a weighted average of luma and color
-v2.1: modified SMAA luma edge detection to dynamically select either red or green as primary luma instead of using fixed luma weights
-v2.1: clamped output of each AA method to prevent rounding error artifacts (generally very rare, but may improve results when handling exceptionally bright scenes)
Last edit: 2 years 4 months ago by lordbean.

Please Log in or Create an account to join the conversation.

  • Martigen
More
2 years 4 months ago - 2 years 4 months ago #24 by Martigen Replied by Martigen on topic (H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)
Hey lordbean,

I've been waiting to try this, I copied previous versions in but the clashes with SMAA/FXAA and being short on time and using a global shared directory for Reshade shaders, I really didn't want to fluff around with cutting out header files etc so haven't actually seen this working yet -- and so the combined 2.1 release incorporating the head files is a good idea! So I tried it out.

It works.. maybe? I can alter HQAA settings and it's definitely doing something on screen, but I get Reshade errors saying AreaTex and SearchTex cannot be found for HQAA. If I remove HQAA.fx from the shader directory, these disappear and SMAA and FXAA work fine. It seems there is still some kind of clash going on.

In case it helps, as above, I use a global repository for Reshade shaders and texture files -- that is, I have one install of these and when I install Reshade to a game, I just copy the .dll and the .ini file that points to these directories. This way I have one set of shaders and texture files for all games on the system, instead of dozens of separate installations. The directories are specified in Reshade's settings and so far all shaders and configs work fine with this.
Last edit: 2 years 4 months ago by Martigen.

Please Log in or Create an account to join the conversation.

  • lordbean
  • Topic Author
More
2 years 4 months ago #25 by lordbean Replied by lordbean on topic (H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)
As long as you've configured the texture search directories correctly, it should be working, so it's hard to diagnose without seeing the exact setup. In theory, there should be no way for SMAA to be able to find the search patterns and HQAA not to in the same install because they both reference the exact same two files (I haven't done anything to that particular piece of code). The SMAA portion of HQAA definitely won't work properly without them though...

Please Log in or Create an account to join the conversation.

  • lordbean
  • Topic Author
More
2 years 4 months ago - 2 years 4 months ago #26 by lordbean Replied by lordbean on topic (H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)
v2.2 is now available. Changed the FXAA setup to determine number of coarse passes to run based on framebuffer height (runs one coarse pass at 1440p, two at 2160p, none at 1080p or lower). This should keep the blur to a minimal level on 1080p screens.

Edit - v2.2.1 pushed. SMAA edge detection settings recalibrated taking advantage of the new adaptive luma detection code, resulting in less blur and better retention of details in the original scene.
Last edit: 2 years 4 months ago by lordbean.

Please Log in or Create an account to join the conversation.

  • Martigen
More
2 years 4 months ago - 2 years 4 months ago #27 by Martigen Replied by Martigen on topic (H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)
That's what I would imagine, but just tried again with a different game using 2.3 (noticed it was just uploaded!). This is the error in the log file:
11:06:24:740 [13720] | INFO | Loading image files for textures ...
11:07:19:814 [08908] | INFO | Successfully loaded "M:\Reshade\Shaders\AdaptiveSharpen.fx".
11:07:19:817 [11788] | INFO | Successfully loaded "M:\Reshade\Shaders\Clarity2.fx".
11:07:19:825 [00824] | INFO | Successfully loaded "M:\Reshade\Shaders\HQAA.fx".
11:07:19:827 [00032] | INFO | Successfully loaded "M:\Reshade\Shaders\PD80_02_Bloom.fx".
11:07:20:007 [13720] | INFO | Loading image files for textures ...
11:07:20:008 [13720] | ERROR | Source "AreaTex.png" for texture 'V__areaTex' could not be found in any of the texture search paths!
11:07:20:008 [13720] | ERROR | Source "SearchTex.png" for texture 'V__searchTex' could not be found in any of the texture search paths!
This doesn't occur if I select SMAA.fx instead of HQAA.fx. If I choose to load both, understandably I get an error saying both textures had already been defined (expected behavior there).

The relevant portion of the config file:
[GENERAL]
CurrentPreset=0
EffectSearchPaths=m:\reshade\shaders
FontGlobalScale=2.000000
IntermediateCachePath=M:\Temp
NoReloadOnInit=0
PerformanceMode=0
PresetPath=.\ReShadePreset.ini
PresetTransitionDelay=1000
SaveWindowState=1
ScreenshotFormat=1
ScreenshotPath=.
ShowClock=0
ShowFPS=1
SkipLoadingDisabledEffects=1
TechniqueSorting=MXAO,LumaSharpen,AdaptiveSharpen,FXAA,SMAA,Clarity,Vibrance,AmbientLight,DisplayDepth,DPX
TextureSearchPaths=m:\reshade\textures
TutorialProgress=4
Last edit: 2 years 4 months ago by Martigen.

Please Log in or Create an account to join the conversation.

  • lordbean
  • Topic Author
More
2 years 4 months ago - 2 years 4 months ago #28 by lordbean Replied by lordbean on topic (H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)
Hmm, the error about textures already being defined is decidedly unintended, I wanted to make sure HQAA could load beside SMAA. I'll fix it and push an update to the one I just did a minute ago.

Sometimes I feel like I'm iterating versions too fast, but I always have a game open with the shader live and running while I'm messing with it... hard to argue with good results when that's what the game keeps showing me.

Edit: fix is now pushed as version 2.4.1. It uses unique names for all textures, samplers, pixel shaders, and vertex shaders, which should hopefully make it actually 100% compatible with SMAA and FXAA.
Last edit: 2 years 4 months ago by lordbean.

Please Log in or Create an account to join the conversation.

  • Martigen
More
2 years 4 months ago - 2 years 4 months ago #29 by Martigen Replied by Martigen on topic (H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)
Oh, well glad I mentioned that then smiley

Tried out 2.4.1, same issue:
15:10:13:298 [11376] | INFO | Successfully loaded "M:\Reshade\Shaders\AdaptiveSharpen.fx".
15:10:13:300 [14800] | INFO | Successfully loaded "M:\Reshade\Shaders\Clarity2.fx".
15:10:13:303 [12376] | INFO | Successfully loaded "M:\Reshade\Shaders\PD80_02_Bloom.fx".
15:10:13:311 [09796] | INFO | Successfully loaded "M:\Reshade\Shaders\SMAA.fx".
15:10:13:312 [10764] | INFO | Successfully loaded "M:\Reshade\Shaders\HQAA.fx".
15:10:13:521 [11896] | INFO | Loading image files for textures ...
15:10:13:536 [11896] | ERROR | Source "AreaTex.png" for texture 'V__HQAAareaTex' could not be found in any of the texture search paths!
15:10:13:547 [11896] | ERROR | Source "SearchTex.png" for texture 'V__HQAAsearchTex' could not be found in any of the texture search paths!
Note loading both SMAA and HQAA given what you said above. Only loading HQAA gives the same error.

I also noticed that ASSMA.fx was complaining -- I think that's your shader too? -- so I removed it from the shader directory. No dice unfortunately, the above errors are with ASSMA.fx removed. When it was in the shader directory, I got twice the number of complaints.

Is it worth copying the .png texture files to reference your own versions?

EDIT: I just tested giving the game its own shader and texture repository, and copying only SMAA, FXAA and HQAA across (and the associated textures to the textures dir). Same error. I then removed SMAA and FXAA but kept their .fxh header files present. Same error. Then I removed the header files as a test (I presume these are needed though) so HQAA is the only shader present now (note this means only HQAA.fx, Resahde.fxh, and ReshadeUI.fxh are in the shader directory). Same error. :0
 
Last edit: 2 years 4 months ago by Martigen.

Please Log in or Create an account to join the conversation.

  • Martigen
More
2 years 4 months ago - 2 years 4 months ago #30 by Martigen Replied by Martigen on topic (H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)
Update -- before you make any further changes.

My shader respository has been built over years, and while I do regularly keep it up to date, it includes shaders from lots of sources in the one directory including shaders not in any of the official repositories. So I decided to do a test with a clean install of Reshade to a game and download all of (and only) those shaders the installer provides, then copied HQAA across -- and it works with no errors.

So it's something particular to my Shaders directory, perhaps another shader is causing this even without being loaded. I'll narrow it down to find what's going on, but don't make any further changes on my account unless it's something you want to do anyway.
Last edit: 2 years 4 months ago by Martigen.

Please Log in or Create an account to join the conversation.

  • lordbean
  • Topic Author
More
2 years 4 months ago - 2 years 4 months ago #31 by lordbean Replied by lordbean on topic (H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)
HQAA doesn't actually need the headers anymore, I integrated them into the actual shader to make modifications easier to do, and to increase compatibility with loading other similar shaders. I honestly have no idea what could be causing this issue given SMAA can find the texture files. All I can think of to suggest at this point is to try doing a localized install to a game folder with the actual executable - but there's no reason this should be any different from your setup, since I do something similar. I keep a ReShade folder on my desktop and although I use the installer to place it with games, I skip the shader download part and then edit the settings to point to the folder on the desktop where I keep all my shaders and supporting resources.

Edit: just tried it myself to make sure I didn't do something that caused a weird problem. Installed it to a random game, placed a checkmark in the SweetFX package to pull the textures, and then dropped HQAA.fx in the shaders folder.
Log


2nd edit - I'm glad it's some sort of conflict other than the actual shader, that would have given me nightmares trying to sort out. I do think you pointed out a real flaw with my attempt at compatibility above though, the fact that it was naming and referencing the textures internally using the stock SMAA and FXAA code was likely going to cause hiccups for someone at some point.
Last edit: 2 years 4 months ago by lordbean.

Please Log in or Create an account to join the conversation.

  • Martigen
More
2 years 4 months ago #32 by Martigen Replied by Martigen on topic (H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)
So if reading again my post 2nd last post one might assume the culprit might be Reshade header files, however since I've managed to achieve a stable installation of the official repositories + my extra shaders, copied across and not replacing to just get the differences, and everything appears to be working fine (including HQAA!) I'm not inclined to do more A/B testing atm :P Bottom line, it's working well :)

Now for some feedback: I really like how it picks up on edges and smoothes them out where SMAA wouldn't (and which was always an advantage of FXAA I found) so congrats there. It definitely has an impact on UI text, particularly if it's small but I don't think there's much to be done there. It's quite soft and definitely needs a sharper after it.

However, an idea -- and I may be speaking out of my ass since I am not a shader developer -- but I've seen some shaders blend the result of their work with a scene based off a strength slider. Currently the two tunables cover edge detection level and subpixels, but is there merit in being able to control how strong the overall effect is by blending? It might allow to smooth edges like those SMAA misses (with a tradeoff based on strength) while minimising impact on text. Or not, again, my ass is talking.

Please Log in or Create an account to join the conversation.

  • lordbean
  • Topic Author
More
2 years 4 months ago - 2 years 4 months ago #33 by lordbean Replied by lordbean on topic (H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)
An interpolate between original scene and resulting scene is possible to implement, yes. You could also try reducing the subpixel effect strength - it pretty directly affects how blurry the resulting image is going to look.

Edit: I'll try fiddling with the inline sharpen I use in the FXAA pass(es) also. I set it up to be fairly conservative, but there might be room to make it stronger since it only affects output from FXAA.

Yet more edit: Please don't be shy with ideas, you don't necessarily need to understand the code for an idea to be good. I welcome feedback both positive and negative as long as it's constructive.

I posted a forked experimental version with a completely different method of calculating the sharpening amount used by the FXAA inline sharpen, find it here . The results look somewhat better to me in my own testing, but the changes are very minor at best, so I'm going to need feedback to decide whether to tweak it further, implement as-is, or just go back to the old method.

Updated the experimental fork since last edit. New experimental version computes sharpening amount based on perceived luminance of the two color channels not selected to represent luma during FXAA edge detection and correction. Results look good to me, it seems to interact less with fine UI elements, leaving them more readable.

Updated experimental fork again (v2.5 this time). This one will likely merge to main branch. I realized the new calculation could return a negative value on very dark pixels so I clamped it to positive numbers. Since this is likely to happen a lot in dark scenes I also implemented an early exit path in FXAA which skips the sharpening calculations entirely if calculated sharpening requirement = 0. Didn't think it'd amount to much but I noted an execution time drop of about 0.1ms on the GPU after the changes, which is more than I expected.
Last edit: 2 years 4 months ago by lordbean.

Please Log in or Create an account to join the conversation.

  • lordbean
  • Topic Author
More
2 years 4 months ago #34 by lordbean Replied by lordbean on topic (H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)
v2.6 is up. I discovered my adaptive luma detection method was making SMAA really sensitive and it was actually responsible for a lot of extra blurring. It has a very high success rate picking up true edges though, so I corrected it by careful re-adjustment of the detection settings until I got it balanced decently.

Please Log in or Create an account to join the conversation.

  • NoPippinNo
More
2 years 4 months ago - 2 years 4 months ago #35 by NoPippinNo Replied by NoPippinNo on topic (H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)
Hello! Thank you for working on AA methods, it's one of the things that drive me crazy the most.

On the subject of text artifacting, I accidentally noticed that something like a very sharp Bumpmapping shader before HQAA cleaned texts up a fair bit for me, though admittedly it probably varies with game fonts and such.

Is there such a thing as a text reinforcement shader out there? (the advantage of a separate shader being hotkeys)

I remember another AA shader having an option using specific color regions to avoid processing text but maybe reinforcing it would yield better results?

I'm a total layman on the inner workings of shaders so if you already do something like this internally please disregard this >.<
Last edit: 2 years 4 months ago by NoPippinNo.
The following user(s) said Thank You: lordbean

Please Log in or Create an account to join the conversation.

  • lordbean
  • Topic Author
More
2 years 4 months ago - 2 years 4 months ago #36 by lordbean Replied by lordbean on topic (H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)
It's unfortunately pretty much impossible to stop a post process AA shader from messing with fonts at least a little bit without being able to run the shader before the game actually draws the text (this is how the built-in AA in most games avoids the problem). One of my priorities is to keep text readable after the shader runs though since I use it as a replacement for TAA in a lot of newer games.

Edit: you got me looking at it a bit to see if I could improve it and I ended up devising a new method of calculating the inline sharpening for the FXAA routine. It interacts a lot more cleanly with UI elements and fine details than I'd managed to make it do before. Pushed it as v2.8.2.
Last edit: 2 years 4 months ago by lordbean.

Please Log in or Create an account to join the conversation.

  • NoPippinNo
More
2 years 4 months ago - 2 years 4 months ago #37 by NoPippinNo Replied by NoPippinNo on topic (H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)
Awesome! I'll test it out in a bit!

I wonder if a narrow range around specific HSV values could be subdued/excluded from the processing?
Font colors might be something fairly easy to mod in many games, and some particularly hard-to-read HSV values can be very specific to "vanilla" text like colors used on loot rarity in various games

If a preset maker had a few fields to input those values on you could offer an extremely accurate filter for almost any game!

In the modded font scenario even what used to be black and white fonts could be slightly tuned with precise hues for filtering XD
Last edit: 2 years 4 months ago by NoPippinNo.
The following user(s) said Thank You: lordbean

Please Log in or Create an account to join the conversation.

  • lordbean
  • Topic Author
More
2 years 4 months ago #38 by lordbean Replied by lordbean on topic (H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)
v2.9 now up. Improved the sharpening amount calculation a bit more (it's walking a pretty fine line between controlling blur and generating artifacts) and implemented baked in quality presets to make it easy to jump in and start using HQAA. Old settings still available when Custom is selected.

Please Log in or Create an account to join the conversation.

  • Martigen
More
2 years 4 months ago #39 by Martigen Replied by Martigen on topic (H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)
Played briefly with v2.9 and really like it, the presets are a great idea! Following up with something like Advanced Sharpening it's looking good.

On the presets, not sure how hover tips work, but is it possible to list the settings when you hover over each preset? I noticed also that Custom, along with the defaults, looks different from all the presets so we technically have five presets to choose from at the moment and it'd be good to see what values they're using to better understand how to tune it.
The following user(s) said Thank You: lordbean

Please Log in or Create an account to join the conversation.

  • lordbean
  • Topic Author
More
2 years 4 months ago - 2 years 4 months ago #40 by lordbean Replied by lordbean on topic (H)ybrid high-(Q)uality (A)nti-(A)liasing (HQAA)
Custom is actually the same as the High preset by default (though if you've used a previous version with a different set of defaults, your ReShade profile may still have those values saved). On a fresh install or after resetting all values though, High and Custom do the same thing. The reference values for the presets are actually pretty easy to find inside the .fx file - they're defined as static constant arrays as the final piece of UI code just underneath the comments at the start of the file.

The actual presets follow a similar mindset to the ones that can be triggered by a #define in SMAA - even the Ultra preset still isn't as sensitive or aggressive as the shader can go, especially if you enable Virtual Photography mode.

That reminds me - v2.9 had a change to Virtual Photo mode I forgot to list. When enabled, Virtual Photo mode disables FXAA's inline sharpen entirely so that there's no artifacts generated by it. The math I'm doing for the sharpen amount is working fairly well in probably 95-98% of scenarios, but it produces the odd artifact now and then on the remaining couple percent.
Last edit: 2 years 4 months ago by lordbean.

Please Log in or Create an account to join the conversation.

We use cookies
We use cookies on our website. Some of them are essential for the operation of the forum. You can decide for yourself whether you want to allow cookies or not. Please note that if you reject them, you may not be able to use all the functionalities of the site.
Ok Decline
More information