So..TAA with SMAA possible with shaders? Seems so!

This just posted:

github.com/aliasIsolation/aliasIsolation/releases

Adds Temporal AA to Alien Isolation's SMAA 1X via shader injection. Can this TAA shader be ported to Reshade?

*gives offering of beer and pizza to Crosire The Great and the Shader Authors of Awesomo Land*

heres some more info:
bitbucket.org/Unity-Technologies/cinemat...e-effects/branch/taa


pd: funny I was asking about TAA just a few days ago.... reshade.me/forum/shader-presentation/223...itty-s-shaders#16421

It seems to be pretty specific for Alien Isolation as far as I can tell, relies on the games own SMAA 1x anti-aliasing and then hooks into D3D11 and tampers with some of the games shaders but eh maybe there's some chance of having it portable but I am unsure if it would work via a generic injector without any game specific compatibility implementations but it would be nice to be able to make use of some form of temporal AA.
(Either via SMAA T2X / 4X or some custom solution as seen in a few newer games.)

From the readme:

The mod works by injecting itself into the executable, and hijacking D3D11 calls. It replaces a few shaders, and injects some of its own rendering.

Has the shaders here but I think it's in compiled hlsl form.
github.com/aliasIsolation/aliasIsolation.../master/data/shaders

EDIT: Ah there's source files too.
github.com/aliasIsolation/aliasIsolation...ster/src/dll/taa.cpp
github.com/aliasIsolation/aliasIsolation...master/src/dll/taa.h
(There's the TAA specific code but there also looks to be a lot of code specific to the game itself so well I have my doubts if it could be ported but I'm no expert.)

It uses compute shaders. ReShade does not support compute shaders (because you need at least D3D11 for those).

crosire wrote: It uses compute shaders. ReShade does not support compute shaders (because you need at least D3D11 for those).

*lower lip wobbles, tears well in eyes*

Can... can there be Reshade shaders that only work with D3D11, clearly specified with the shader? And and and... support built-in to reshade to handle compute shaders?

Or can this compute shader version be ported to, um, the other shader things we use. The ones that aren't compute. *shakes fist at compute*

(I realise I know not the volume of work that I ask, so slap me a reality check if necessary)

Also Edit: Since I can't talk programmer -- maybe someone here who does can approach the author and see if they want try and port it to Reshade. He clearly has talent and may be interested in making shaders for Reshade, even if in general.

This may be a little bit offtopic, but I've been reading since a long time in this forum about "fake downsampling" by ScreenSize*2 (or so) and then apply SMAA or FXAA but all the times I tried such experiments I failed, and when I asked for examples, or if anyone actually made it I get no answer... so is there any REAL possibility to improve antialiasing over the current methods we already have in Reshade?

Screenshots with TAA on/off in AI:

forums.guru3d.com/showpost.php?p=5330228&postcount=712

That's crazy good!

Martigen wrote: Can this TAA shader be ported to Reshade?

The shader itself is nothing special, and could easily be ported. It wouldn't really need to be compute either. The particular implementation in Alias Isolation is DXBC (compiled) rather than HLSL, but the one from Unity or Inside could be used instead.

What's more problematic is the setup around the shader.

In the Alien mod, the TAA pass runs in a specific place in the frame, before motion blur or bloom gets calculated. This means that those effects remain alias-free. I'm not aware of any method one could use to identify such a place in the frame in a generic way; it would require game-specific code.

Then there's the scene jittering. I accomplish it by hooking constant buffer upload functions, and modifying transformation matrices passed from the CPU to the GPU. That should only be done for full-res render passes which run before the Temporal AA shader. It should be possible to guess by looking at the currently bound render target, but one could end up touching other passes anyway -- e.g. shadow map rendering. If you jitter those, you will end up with flickering in shadows. On top of that, you need to know which fields to manipulate in the constant buffers. Alien Isolation makes it pretty easy by sharing constant buffer layouts between passes, but this is not the case with every game out there. One could run TAA without jittering -- it would then give you firefly suppression, but wouldn't have the nice super-sampled look.

One more thing which TAA requires is velocity vectors. They are critical for the 'temporal' part of the algorithm, and allow you to find the corresponding pixel in the previous frame for any pixel in the current. Without them, TAA results in a world of ghosting and swimming pixels. Alien renders pretty decent velocity vectors for its SMAA and motion blur, so it was just a matter of finding them among the resources the game creates. There's an alternative here: one could take the depth buffer, the current and previous transformation matrices (assuming those can be identified), and then generate the velocity vectors assuming the scene is static. Any dynamic objects would then look smeared, but at least the static parts would be fine.

In the case of Alien, I also needed to kill the barrel distortion and modify chromatic aberration rendering, as they were messing up with the image,.TAA then wouldn't be able to fix it completely. I would take a capture with RenderDoc, identify an offending shader, note down its GUID, and then intercept its creation on the render device in code. With that, I could replace it with an alternative.

One could probably create a bunch of heuristics to identify the above components, and try to do it in a generic way, but to have a solid TAA implementation, game-specific code would almost definitely be required.

May I ask if the jittering always has to introduce some amount of pixel crawling?
I've noticed it with every game that offers such a TAA solution (especially Doom, but also Frostbite, Unreal Engine, Cryengine, Fallout 4).
I wonder if Uncharted 4 has this problem as well?
Cryengine offers to turn it off, which leaves quite some jaggies of still image.

May I ask if the jittering always has to introduce some amount of pixel crawling?

It doesn't have to, but it's very difficult to avoid it entirely. It can be caused by issues with the TAA shader, but also by tricky content.

The way triangle rasterization works in GPUs is that a pixel is shaded when its center is enclosed by the triangle. If you have a very small feature, say a lamp post on the horizon, it can be easily missed if it doesn't happen to cover any pixel center. The jittering in TAA will cause small features to sometimes hit pixel centers, and sometimes miss them. It then becomes almost impossible to do something sensible with the flickering pixels. One could blend them with the corresponding location in the previous frame, but if it's a tiny feature moving in parallax with the background, there might not be a corresponding location. And sometimes you don't want to blend them at all. What if a light turned on in the new frame? You don't want to smoothly blend it with the previous one; it should appear right away.

Another source of individual flickering pixels is shiny surfaces. If you have a point light and an almost perfectly smooth surface, a pixel will need to be 'lucky' to be facing in just the right direction in order to catch the specular highlight. If it's just off the perfect angle, it doesn't catch the light. When you have a shiny object which moves or rotates in relation to the camera, this situation happens all the time. It also happens with highly tessellated meshes, or due to shader aliasing, because tiny features with different orientations can end up in the center of each pixel on the screen.

The way TAA tries to solve the issue is by looking at a small neighborhood around each pixel (typically 3x3 pixels), and deciding that the colors in that small tile are 'valid' colors, and anything vastly different is aliasing. It then compares the corresponding color in the previous frame to the 'valid' colors, and either accepts the reprojected color if it looks okay, or adjusts it otherwise. That way if the 3x3 pixels in the current frame are yellow (because they're on a duckling), and the reprojected pixel is also yellow-ish, TAA figures it can blend the values together. But if the previous pixel was green, the duckling must have just materialized here, and the green color must be discarded. This is typically done by building a bounding box over the R G and B color components in the small neighborhood; that is, you find the lowest and highest value of each component in the 3x3 pixel set, and then anything which is between the min and max for all of R, G, and B, is a valid color.

Flickering pixels pose a problem to the simple color-based oracle. If by chance a bright pixel appears in a few frames in the same spot, but then (again, by chance) disappears, the color similarity test will suddenly say "oh, but this looks wrong. according to my 3x3 pixel tile, there shouldn't be any bright pixels here, but the previous frame had one". Then it will kill the bright highlight which the player has been seeing for a few frames. Now if in the next frame there are a few bright pixels, they will start fading in, until the test kills them again. And so on, this can repeat itself in cyclic patterns.

Several heuristics can be employed to try and reduce pixel crawling (or flickering), Brian Karis suggested one in the UE4 Temporal AA presentation at Siggraph. It's pretty effective, but it doesn't completely solve the issue. One could probably come up with an elaborate scheme to solve the issue, but Temporal AA typically doesn't have more than a millisecond in a frame to do its thing because of game rendering performance budgets. So we end up with crawlies. At least until the rendering industry figures out a new way of dealing with them

A joy to read, thanks for the explanation.
Despite of the pixel crawling artifacts, I still think that TAA is one of the most exciting inventions for realtime graphics. I'd also say that UE TAA probably suffers the least by it, at least from judging the result of Unreal Tournament with UE 4.12, while it's worst in Doom (TSSAA 8TX, it also leaves some jaggies and can introduce nasty ghosting with some textures).

A shame that so many games are still not offering any temporal AA at all, e.g. Forza Apex. Devs seem to think that with forward rendering, they just need MSAA and everything's alright. It certainly isn't, I'd choose 2xMSAA + SMAA TX anytime over just plain 8xMSAA...
And of course too bad that TAA isn't easy to inject.

I guess some one could port PlayDead Temporal AA that uses reprojection.
If you really want Temporal AA that bad .....
github.com/playdeadgames/temporal

mindu wrote: This may be a little bit offtopic, but I've been reading since a long time in this forum about "fake downsampling" by ScreenSize*2 (or so) and then apply SMAA or FXAA but all the times I tried such experiments I failed, and when I asked for examples, or if anyone actually made it I get no answer... so is there any REAL possibility to improve antialiasing over the current methods we already have in Reshade?

I'm also interested!
If i'm not wrong FXAA 4 never released by thimoty lottes was based on this idea!!!
"FXAA 4 is described as "a combination of a demosaicing/deinterleaving, super-resolution, and anti-aliasing algorithm all in one tiny full screen pass."
www.neoseeker.com/news/17931-fxaa-40-in-...-stage/#&gid=1&pid=1

So I ported this TAA code. Does not work to well. It may need some work. But, it's simple. Feels like it needs to be used with a other form of AA.

To me it feels more like a near sight simulator.

And where is the "temporal" part???

Marty McFly wrote: And where is the "temporal" part???

Ya it's not very complete. I just ported his implantation. All of his information and where I got it, is in the shader it self.

Basically missing this part.
www.slideshare.net/TiagoAlexSousa/antial...thods-in-cryengine-3
I think this links to the right slide.

There are some aspects there though. I was just reading this out line for unreal TAA.
de45xmedrsdbp.cloudfront.net/Resources/f...A_small-59732822.pdf

I also need to translate his notes since it's not in English.

Any one here know a game that has bad crawling and flickering with the camera is in motion?

BlueSkyKnight wrote: Any one here know a game that has bad crawling and flickering with the camera is in motion?

Pick any PBR game and turn off any in-game antialiasing.

MaxG3D wrote:

BlueSkyKnight wrote: Any one here know a game that has bad crawling and flickering with the camera is in motion?

Pick any PBR game and turn off any in-game antialiasing.

I don't know what PBR stands for.

Physically Based Rendering (Sometimes there's also Physically based shading.) many newer games implement (Usually simpler.) variations of this technique to make materials look more distinct IE in Witcher 3 you can tell the different materials on Geralts armor apart fairly easily.

Implementation varies a bit from one game to another, Fallout 4 for example uses a somewhat different method so you can have some materials or areas that look extremely good and then some others that are more bland.
(Materials are only one part of it though, accurate lighting and shadowing are also important but also usually very expensive in terms of performance.)

Better implementation of effects such as specular highlighting ("shine") along with post-process effects for anything from shaders for chroma shifting / chromatic aberration to noise or just bloom, screen-space reflections and color tone adjustments can cause more noticeable aliasing, shader aliasing in particular has been a big problem since MSAA only works on geometry with limitations and for a deferred renderer the performance hit is often significant and SMAA or FXAA are also only partially effective so it takes downsampling or SSAA though temporal AA can also be effective depending on it's quality. (UE4 has a really good implementation and although usually not exposed to the in-game options in games it is actually very customizable.)