FXAA & SMAA - only partially removes aliasing

FXAA
I always assumed that ReShade FXAA was superior to NVidia FXAA and any games' built-in FXAA. However, I have reconsidered because when I use default ReShade FXAA settings with the default or even reduced Threshold value, a lot of aliasing stays, which I thought was fine, BUT then I made comparison between in-game FXAA (Killing Floor 2 and Dying Light) vs. ReShade FXAA and here's what I found:
- In-game FXAA removes aliasing that ReShade FXAA does not touch
- ReShade FXAA removes aliasing that in-game FXAA does not touch

Inquiries:
- Why is it like that? I figured that maybe in-game FXAA uses Sub-Pixel AA, which softens/blurs the image but also removes more aliasing. ReShade FXAA Sub-Pixel AA is set to 0.000 and I have no idea which values I should try and whether they could help at all. NVidia FXAA also uses Sub-Pixel AA. Could that be the cause of why some games' built-in FXAA removes aliasing that ReShade FXAA does not remove ?
- What about NVidia FXAA? It supposedly uses the most advanced algorithms and such, but I am not sure. Is there a comparison somewhere?
- What settings and/or values should I adjust in ReShade FXAA to get the benefits of both - ReShade FXAA and in-game FXAA or at least get the benefits of ReShade FXAA & NVidia FXAA



SMAA
ReShade SMAA is usually miles ahead of any game's built-in SMAA, except for COD - BO III Cinematic SMAA 2x, which removes a lot more aliasing than ReShade SMAA.

Inquiries:
- Is ReShade SMAA considered to be equivalent to 1x SMAA or 2x SMAA setting that I often see in games ?
- Which settings / values should I adjust to get the same Cinematic 2x SMAA effect as in COD - BO III?
- Which settings and/or values should I adjust in general to make SMAA remove more aliasing? I already lowered the Threshold to the minimum of 0.05.

ReShade shader framework SMAA should be the "standard" 1x version as it's often called though it's possible to customize the image quality of it instead of just relying on presets.
(Most games probably use performance over quality thus ReShade SMAA being better.)

It can't do temporal AA though as that requires access to well I forgot what exactly it was called but ReShade the injector can't easily detect and use this in a generic way as I understood it. (Velocity something?)

ReShade can't do MSAA either so 2x or 4x SMAA which utilizes MSAA in addition to SMAA and temporal if it's T2X or 4X is unavailable too.
(SLI/Crossfire friendly SMAA via SMAA 2XS is probably even more difficult.)

For FXAA I don't know much about it as I personally prefer SMAA but it should be the latest public version of 3.11 and again more easily customized than what's offered in most games.


EDIT: As far as I know enabling both FXAA and SMAA is a bit of a waste and generally will just blur the image for a minor reduction in aliasing, probably the same if you use in-game post-AA with injected post-AA.

MonarchX wrote: FXAA
I always assumed that ReShade FXAA was superior to NVidia FXAA and any games' built-in FXAA. However, I have reconsidered because when I use default ReShade FXAA settings with the default or even reduced Threshold value, a lot of aliasing stays, which I thought was fine, BUT then I made comparison between in-game FXAA (Killing Floor 2 and Dying Light) vs. ReShade FXAA and here's what I found:
- In-game FXAA removes aliasing that ReShade FXAA does not touch
- ReShade FXAA removes aliasing that in-game FXAA does not touch

Inquiries:
- Why is it like that? I figured that maybe in-game FXAA uses Sub-Pixel AA, which softens/blurs the image but also removes more aliasing. ReShade FXAA Sub-Pixel AA is set to 0.000 and I have no idea which values I should try and whether they could help at all. NVidia FXAA also uses Sub-Pixel AA. Could that be the cause of why some games' built-in FXAA removes aliasing that ReShade FXAA does not remove ?
- What about NVidia FXAA? It supposedly uses the most advanced algorithms and such, but I am not sure. Is there a comparison somewhere?
- What settings and/or values should I adjust in ReShade FXAA to get the benefits of both - ReShade FXAA and in-game FXAA or at least get the benefits of ReShade FXAA & NVidia FXAA



SMAA
ReShade SMAA is usually miles ahead of any game's built-in SMAA, except for COD - BO III Cinematic SMAA 2x, which removes a lot more aliasing than ReShade SMAA.

Inquiries:
- Is ReShade SMAA considered to be equivalent to 1x SMAA or 2x SMAA setting that I often see in games ?
- Which settings / values should I adjust to get the same Cinematic 2x SMAA effect as in COD - BO III?
- Which settings and/or values should I adjust in general to make SMAA remove more aliasing? I already lowered the Threshold to the minimum of 0.05.

all cinematic and 2x+ smaa in cod games are temporal aa either with msaa or supersampling and is a much more refined version then whats publicly available. Driver fxaa will almost always be best to use, but smaa/fxaa on reshade are decent enough and generally better then their counterparts in games graphic menus. As to settings read inside the actual code for the files and they give you suggested values for quality/performance

SMAA T2x requires moving camera in sub pixel jitters.
(Not just temporal, with sub-pixel jitter, the AA will get more detail then per-pixel temporal AA and also work in static scene. Btw, the jitter is also the cause of flickering in Fallout 4 even with console up". Also the full T2x will require velocity buffer to weight between samples.)

SMAA S2x requires MSAA 2x buffer

SMAA 4x requires both of above

(Edit: on FXAA)
The difference in FXAA could be difference in settings, the header implementation is quite straight forward. However the FXAA itself can also combined with multisample buffer and sub-pixel jitter as shown in NV presentation on SIGGRAPH2011.

(Edit again: after checking the port & FXAA header)
[strike]Ceejay stated he's using green channel as luminence in his file, which can affect the edge detection. [/strike]
Nope, Ceejay had modified the FXAA header to the original one by Timothy Lottes..... RGB is used to calculate luminence. Small detail though, there should be a additional pow(lum, 1/2.2) on luminence to perform FXAA in perceptual space.


A demo from SMAA document

GTA V, Killing Floor 2 and Dying Light have shown to be different from other games. In at least these 3 games:
- In-game FXAA removes most aliasing and removes aliasing than ReShade FXAA does not
- ReShade FXAA (unedited values) takes the second spot and removes aliasing that in-game FXAA does not
- NVidia FXAA blurs image the most and does not remove most aliasing <--- this is common to ALL games, not just the 3 listed above. NVidia FXAA SUCKS! I would love to check out some true comparison shots of NVidia FXAA vs. any other FXAA.

HUGE thanks on pointing to the actual Shader files! They told me what I needed to get the best image for FXAA, but SMAA Advanced and Predication Settings are not explained in terms of their effect on image quality, although all I wanted to know if the difference in quality between SMAA Edge Detection values (1 or 2 or 3), where 1 = Luma Edge Detection, 2 = Color Edge Detection, 3 = Depth Edge Detection.

What exactly is Sub-Pixel and Sub-Pixel AA? I know Sub-Pixel AA BLURS the image, but does that blurring actually remove aliasing more than it would with Sub-Pixel AA turned off or does it just make overall image softer without removing any additional? Googling it didn't provide a satisfactory and simple-enough answer...


Off-Topic:
How does TAA work? Its used in Vanishing of Ethan Carter - Redux, Fallout 4, and Tom Clancy's The Division. TAA makes the image blurry, but it removes more aliasing than any other post-processing method used in current games, except for maybe OGSSAA. TAA + LumaSharpen and/or AdaptiveSharpen = almost no aliasing + sharp image. What's most significant is that TAA is not even as heavy as on GPU as 4x MSAA @ 1080p+. It reduces FPS more than FXAA and more than SMAA, I think, but not as much as 4x MSAA and of course not as much OGSSAA, yet the effect is similar to OGSSAA + blur. AFAIK, it does not blur as much as TXAA. Every game should come with TAA!

A quote from SMAA header

* You've three edge detection methods to choose from: luma, color or depth.
* They represent different quality/performance and anti-aliasing/sharpness
* tradeoffs, so our recommendation is for you to choose the one that best
* suits your particular scenario:
*
* - Depth edge detection is usually the fastest but it may miss some edges.
*
* - Luma edge detection is usually more expensive than depth edge detection,
* but catches visible edges that depth edge detection can miss.
*
* - Color edge detection is usually the most expensive one but catches
* chroma-only edges.

SMAA predication requires additional input texture as reference to increase or decrease edge detection threshold dynamically.
(SweetFX default to use scene depth for predication texture)

Sub-pixel is that when the 2d projection is smaller then a single pixel. And since the object or edges is so small, it will popping in & out determine by if the pixel sampling point happened to hit it or not, which is most noticeable on wires and and edge aliasing.

To solve that, we employ super-sampling to have multiple sample points per pixel and average the result, hence the SSAA and MSAA to display the partially filled pixel with a level of transparency.

Now, having multiple sample point would surely cost a huge amount of frame rate, doubling the sample count will put 4 times of loads to the machine, and that is where various AA method comes in.

Some use estimators to reconstruct the sub-pixel elements without adding more sampling point like SMAAx1 and FXAA. While they can handle most edges, they cannot guess what is not on the screen as shown in the comparison pic in previous post.

To compensate that, a additional temporal term, TAA, is introduced (different to motion blur). The idea is, while having 1 sample per pixel, it move around the sample point a little bit within the pixel between frames and average them as if it is multisampling the scene. In the mean time, it is possible to create wrong interpolation if the scene is changing too fast, which is the ghosting artifact on early TAA. As the fix, velocity buffer is used to weight the samples and to avoid blending on complete different edges.

The temporal term could be added to most of the AA method. For example, SMAA x4 = SMAA x1 + MSAA x2 + TAA

btw, OGSSAA, Odered Grid SSAA, is a variation of SSAA that uses different sampling pattern.

kingeric1992 wrote: A quote from SMAA header

* You've three edge detection methods to choose from: luma, color or depth.
* They represent different quality/performance and anti-aliasing/sharpness
* tradeoffs, so our recommendation is for you to choose the one that best
* suits your particular scenario:
*
* - Depth edge detection is usually the fastest but it may miss some edges.
*
* - Luma edge detection is usually more expensive than depth edge detection,
* but catches visible edges that depth edge detection can miss.
*
* - Color edge detection is usually the most expensive one but catches
* chroma-only edges.

SMAA predication requires additional input texture as reference to increase or decrease edge detection threshold dynamically.
(SweetFX default to use scene depth for predication texture)

Sub-pixel is that when the 2d projection is smaller then a single pixel. And since the object or edges is so small, it will popping in & out determine by if the pixel sampling point happened to hit it or not, which is most noticeable on wires and and edge aliasing.

To solve that, we employ super-sampling to have multiple sample points per pixel and average the result, hence the SSAA and MSAA to display the partially filled pixel with a level of transparency.

Now, having multiple sample point would surely cost a huge amount of frame rate, doubling the sample count will put 4 times of loads to the machine, and that is where various AA method comes in.

Some use estimators to reconstruct the sub-pixel elements without adding more sampling point like SMAAx1 and FXAA. While they can handle most edges, they cannot guess what is not on the screen as shown in the comparison pic in previous post.

To compensate that, a additional temporal term, TAA, is introduced (different to motion blur). The idea is, while having 1 sample per pixel, it move around the sample point a little bit within the pixel between frames and average them as if it is multisampling the scene. In the mean time, it is possible to create wrong interpolation if the scene is changing too fast, which is the ghosting artifact on early TAA. As the fix, velocity buffer is used to weight the samples and to avoid blending on complete different edges.

The temporal term could be added to most of the AA method. For example, SMAA x4 = SMAA x1 + MSAA x2 + TAA

btw, OGSSAA, Odered Grid SSAA, is a variation of SSAA that uses different sampling pattern.

How can you have something smaller than a single pixel? Is there a comparison screenshot somewhere that shows how Sub-Sampling improves FXAA? I know in FXAA is heavily blurs the image, but SMAA also uses Sub-Pixel AA and yet it does not blur the image as much as FXAA Sub-Pixel AA. I guess sub-pixel information is internal, like info stored in some buffer that the current edge of a single frame is less than pixel, but in the next frame it will be bigger?

For SMAA, I enabled PREDICATION, which uses for Color and Depth Detection methods at the same time. The DETECTION method is set to 2 (Color), although I am not certain that Chrome edges are more common than Luma edges. Why can't it do Luma + Chroma?

Is it possible to create a generic TAA shader version for ReShade?

FXAA alone doesn't do actual subpixel, it approximate them in screenspace, so does SMAAx1. They estimate subpixels edges through surrounding pixels.

For subpixels and supersampling
This is a vector projected into screen space on a single pixel.
The triangle part is red and background is white.since it can only present one color per pixel, we need to decide which color we want. Normally, the color on the dead center will be chosen.Which is where aliasing is coming from.

but then, if super sampling is used,
and TAA,Temporal AA would require offset on projection matrix that currently not support in reshade.

About Luma + Chroma, it is possible to combine them both with "or" statement (Luma || Chroma). Can't say where it will go though.

FXAA
Yeah, I think I fail to understand even these basics because I can't get past the idea that a pixel can be partially lit... Its either ON or OFF, isn't it? I do understand the decision process you've provided though. There's not a single visual example (actual screenshot, not images made with _____ and ||||) that shows the difference between FXAA without Sub-Pixel AA and FXAA with Sub-Pixel AA. Bottom-line question: Does Sub-Pixel AA sub-setting help SweetFX/CeeJay FXAA setting remove additional aliasing that FXAA without Sub-Pixel AA can't? Its either a Yes or a No .

SMAA
LumaSharpen is applied before SMAA, so if SMAA uses Color + Depth Edge Detection, then it will not smooth-out Luma edges caused by LumaSharpening. Is that correct?

SMAA + MSAA
I noticed that in games, there's never an in-game option to apply SMAA on top of MSAA. You can apply SMAA by itself or apply MSAA by itself or apply MSAA with FXAA, but never MSAA + SMAA. The only reason for that is because MSAA + SMAA make aliasing worse and I agree entirely. Here's more info (closer to the bottom) that proves it - delightlylinux.wordpress.com/2014/02/17/sweetfx-smaa/ . I get the same exact effect when I apply ReShade SMAA on top of any game's MSAA. It makes edges much thicker and while it removes some aliasing that MSAA cannot remove on its own, SMAA also creates some aliasing in other places when used on top of MSAA. Why is that? Are there ReShade SMAA settings that can prevent SMAA from creating additional aliasing when applied on top of MSAA? I also assume that in-game 2x (or 2xT or 2Tx) SMAA, like COD - BO III Film 2x SMAA uses its own implementation of MSAA to make it work with SMAA.

TXAA + FXAA
Assassin's Creed: Syndicate has an in-game option for TXAA + FXAA, but not for TXAA by itself. Why apply FXAA on top of TXAA? Could it be because TXAA doesn't offer Sub-Pixel AA while FXAA does?

Thank you for the effort to explain this to me! I am eager to learn and with your guidance, its not difficult to understand these concepts!

Pixels cannot be partially lit, it mix colors to represent the look.

Here is a quote on FXAA header on subpix settings.

// Only used on FXAA Quality.
// This used to be the FXAA_QUALITY__SUBPIX define.
// It is here now to allow easier tuning.
// Choose the amount of sub-pixel aliasing removal.
// This can effect sharpness.
// 1.00 - upper limit (softer)
// 0.75 - default amount of filtering
// 0.50 - lower limit (sharper, less sub-pixel aliasing removal)
// 0.25 - almost off
// 0.00 - completely off
FxaaFloat fxaaQualitySubpix,

Color and luminance are correlated, they will be affecting each others.

Combining AA techniques does not work like that, it requires modifications and adaptations, not just simply apply them one after another.

SMAA on top of MSAA is called SMAA S2x, which is SMAA perform on multisampled subsamples, then downscale to screen size.
(2 samples per pixel, output to 2 separate rendertargets and perform SMAAx1 on both of them, then blend the outcome)

TXAA provides higher quality then FXAA where TXAA actually do subpixle sampling while FXAA only approximate them.

Judging from NV's description on the technique,

Next-Gen TXAA Anti-Aliasing Makes Its Debut In Latest Update For The Secret World

....in general 2xTXAA offers visual quality on par with 8xMSAA, with performance similar to that of 2xMSAA, and 4xTXAA offers image quality superior to 8xMSAA, with performance comparable to that of 4xMSAA....

it is possible TXAA is something combines TAA and MSAA like this.
TXAA x2 == MSAA x2 + TAA x4 + additional filter
TXAA x4 == MSAA x4 + TAA x4 + additional filter

As the same technique is used on SMAA to achieve SMAA x4, I would say TXAA + FXAA is no differ. FXAA perform on all those additional subsamples TXAA is using, to estimate subpixels in a even smaller grid.

About comparison shots, I believe there are websites that have better illustration on the topic.

Comparison Screenshot is worth a 10000 words of explanation and I think I found it accidentally. The difference between Dark Souls III + no-game AA + FXAA without Sub-Pixel AA is very different from Dark Souls III + no-game AA + FXAA but WITH Sub-Pixel AA. I just wish I could make a screenshot of an identical frame, but I don't think there's a way to pause the game...

In Arma 2&3, SMAA can be applied on top of MSAA:)

acknowledge wrote: In Arma 2&3, SMAA can be applied on top of MSAA:)

Then they must be using some custom code if SMAA can be added on top of MSAA from within game settings. Otherwise, normal MSAA + SMAA(not the custom Filmic 2x SMAA) makes edges very thick and adds some aliasing, while removing other aliasing. MSAA + FXAA looks is miles better. Besides, ReShade SMAA is only good for static angle-games because it pales to faster FXAA when it comes to removal temporal aliasing. Without Sub-Pixel AA, FXAA doesn't blur all that much and it can easily be offset by default LumaSharpen settings + Method 4 (Pyramid) or AdaptiveSharpen.