not enough AA

Hi all! I just became a member to post about this problem.

I recently installed Win X and all my games work just fine except for one. This is a sailing game from 2003 using DX8 called "Pirates of the Caribbean".

On land and in the menus all the action became slow motion like everyone was moving around under water. At sea all is still fine. While searching for a solution someone mentioned Reshade and it works! Kinda sorta........
It does indeed make on land action and menus work at full speed. Yay! But is seems to have removed most of the AA from the action at sea. These old sailing ships have masts and yards and sails and all kinds of ropes going all over the place and the jaggies are very noticeable. In fact the ropes turn into Morse Code. All dots and dashes.
I have been trying to sort it out without much success. SweetFX seems to be the only one that has any AA. I have found that SMAA gives error messages on startup that do not go away. FXAA works except not nearly well enough.
I have watched the tutorial on how to install and set up Reshade and believe it is set up properly. I have read posts about FXAA and think it is set up as high quality as it can go. Is there another setting I am unaware of?
For comparison I have a preset for this game using 24EQ AA and 16X AF and there is almost no jaggies and the ropes are clearly visible.

miklkit50 wrote: I have found that SMAA gives error messages on startup that do not go away

What error messages? Please post them here (they are also written into the log).

The 24x EQAA thing (8x forced MSAA in CCC with Edge-Detect as the method, possibly together with Enhance AA compatibility mode and Adaptive-AA or SSAA over regular MSAA.) makes it sounds like you are forcing AA via the display driver which might be one possibility as to why ReShade is failing as the software is not entirely compatible with forced anti-aliasing, in-game MSAA and such usually work with the caveat that the depth buffer dependent effects in ReShade will usually not work.
(I do not know the technical reason for this however but I guess it's some conflict between DirectX/OpenGL and the driver and then ReShade in some manner.)

Here is a log with the errors. I am unfamiliar with this forum's format and could not find a way to upload the D3D8.log.
01/11/2015 08:34:53:744 [05536] | INFO | Initializing Crosire's ReShade version '1.0.0.915' built on '2015-08-10 01:52:59' loaded from "\POTC4\d3d8.dll" to "\POTC4\POTC4.exe" ...
01/11/2015 08:34:53:806 [05536] | INFO | Registering hooks for "C:\WINDOWS\system32\d3d8.dll" ...
01/11/2015 08:34:53:806 [05536] | INFO | > Delayed.
01/11/2015 08:34:53:806 [05536] | INFO | Registering hooks for "C:\WINDOWS\system32\d3d9.dll" ...
01/11/2015 08:34:53:806 [05536] | INFO | > Delayed.
01/11/2015 08:34:53:806 [05536] | INFO | Registering hooks for "C:\WINDOWS\system32\d3d10.dll" ...
01/11/2015 08:34:53:806 [05536] | INFO | > Delayed.
01/11/2015 08:34:53:806 [05536] | INFO | Registering hooks for "C:\WINDOWS\system32\d3d10_1.dll" ...
01/11/2015 08:34:53:806 [05536] | INFO | > Delayed.
01/11/2015 08:34:53:806 [05536] | INFO | Registering hooks for "C:\WINDOWS\system32\d3d11.dll" ...
01/11/2015 08:34:53:806 [05536] | INFO | > Delayed.
01/11/2015 08:34:53:806 [05536] | INFO | Registering hooks for "C:\WINDOWS\system32\dxgi.dll" ...
01/11/2015 08:34:53:806 [05536] | INFO | > Delayed.
01/11/2015 08:34:53:806 [05536] | INFO | Registering hooks for "C:\WINDOWS\system32\opengl32.dll" ...
01/11/2015 08:34:53:806 [05536] | INFO | > Delayed.
01/11/2015 08:34:53:806 [05536] | INFO | Registering hooks for "C:\WINDOWS\system32\user32.dll" ...
01/11/2015 08:34:53:806 [05536] | INFO | > Libraries loaded.
01/11/2015 08:34:53:822 [05536] | INFO | > Found 5 match(es). Installing ...
01/11/2015 08:34:53:885 [05536] | INFO | > Installed 5 hook(s).
01/11/2015 08:34:53:885 [05536] | INFO | Registering hooks for "C:\WINDOWS\system32\ws2_32.dll" ...
01/11/2015 08:34:53:885 [05536] | INFO | > Delayed.
01/11/2015 08:34:53:885 [05536] | INFO | Initialized.
01/11/2015 08:34:58:325 [05536] | INFO | Redirecting 'Direct3DCreate8(220)' ...
01/11/2015 08:34:58:325 [05536] | INFO | > Passing on to 'Direct3DCreate9':
01/11/2015 08:34:58:325 [05536] | INFO | Installing delayed hooks for "C:\WINDOWS\system32\d3d9.dll" (Just loaded via 'LoadLibraryA("d3d9.dll")') ...
01/11/2015 08:34:58:325 [05536] | INFO | > Found 9 match(es). Installing ...
01/11/2015 08:34:58:466 [05536] | INFO | > Installed 9 hook(s).
01/11/2015 08:34:58:466 [05536] | INFO | Redirecting 'Direct3DCreate9(32)' ...
01/11/2015 08:34:58:466 [05536] | INFO | Installing delayed hooks for "C:\WINDOWS\system32\d3d8.dll" ...
01/11/2015 08:34:58:466 [05536] | INFO | > Found 1 match(es). Installing ...
01/11/2015 08:34:58:466 [05536] | INFO | > Installed 1 hook(s).
01/11/2015 08:34:58:482 [05536] | INFO | Redirecting 'IDirect3D8::CreateDevice(00E6CDC0, 0, 1, 0001078C, 128, 00D3EC34, 065033BC)' ...
01/11/2015 08:34:58:482 [05536] | INFO | Redirecting 'IDirect3D9::CreateDevice(00D818E0, 0, 1, 0001078C, 0x80, 00D3EBAC, 00D3EBA0)' ...
01/11/2015 08:34:58:607 [05536] | INFO | Recreated effect environment on runtime 0110DF30.
01/11/2015 08:34:58:607 [05536] | INFO | Redirecting 'IDirect3DDevice8::CreateVertexShader(00F3F510, 06548400, 0654B1E8, 06548CB4, 0)' ...
01/11/2015 08:34:58:607 [05536] | INFO | > Translating vertex declaration ...
01/11/2015 08:34:58:607 [05536] | INFO | > Disassembling shader and translating assembly to Direct3D 9 compatible code ...
01/11/2015 08:34:58:622 [05536] | INFO | > Replacing version 'vs_1_0' with 'vs_1_1' ...
01/11/2015 08:34:58:622 [05536] | INFO | Redirecting 'IDirect3DDevice8::CreatePixelShader(00F3F510, 0654AEF0, 06548CCC)' ...
01/11/2015 08:34:58:622 [05536] | INFO | > Disassembling shader and translating assembly to Direct3D 9 compatible code ...
01/11/2015 08:34:58:622 [05536] | INFO | Redirecting 'IDirect3DDevice8::CreateVertexShader(00F3F510, 065484B8, 0654B758, 06548C6C, 0)' ...
01/11/2015 08:34:58:622 [05536] | INFO | > Translating vertex declaration ...
01/11/2015 08:34:58:622 [05536] | INFO | > Disassembling shader and translating assembly to Direct3D 9 compatible code ...
01/11/2015 08:34:58:622 [05536] | INFO | > Replacing version 'vs_1_0' with 'vs_1_1' ...
01/11/2015 08:34:58:638 [05536] | INFO | Redirecting 'IDirect3DDevice8::CreatePixelShader(00F3F510, 0654B018, 06548C84)' ...
01/11/2015 08:34:58:638 [05536] | INFO | > Disassembling shader and translating assembly to Direct3D 9 compatible code ...
01/11/2015 08:34:58:638 [05536] | INFO | Redirecting 'IDirect3DDevice8::CreatePixelShader(00F3F510, 0654B018, 06548C9C)' ...
01/11/2015 08:34:58:638 [05536] | INFO | > Disassembling shader and translating assembly to Direct3D 9 compatible code ...
01/11/2015 08:34:58:638 [05536] | INFO | Redirecting 'IDirect3DDevice8::CreateVertexShader(00F3F510, 065484F0, 0654A338, 06548CE4, 0)' ...
01/11/2015 08:34:58:638 [05536] | INFO | > Translating vertex declaration ...
01/11/2015 08:34:58:638 [05536] | INFO | > Disassembling shader and translating assembly to Direct3D 9 compatible code ...
01/11/2015 08:34:58:638 [05536] | INFO | > Replacing version 'vs_1_0' with 'vs_1_1' ...
01/11/2015 08:34:58:638 [05536] | INFO | Redirecting 'IDirect3DDevice8::CreatePixelShader(00F3F510, 0654A338, 06548CFC)' ...
01/11/2015 08:34:58:638 [05536] | INFO | > Disassembling shader and translating assembly to Direct3D 9 compatible code ...
01/11/2015 08:34:58:653 [05536] | INFO | Redirecting 'IDirect3DDevice8::CreatePixelShader(00F3F510, 0654A208, 06548D14)' ...
01/11/2015 08:34:58:653 [05536] | INFO | > Disassembling shader and translating assembly to Direct3D 9 compatible code ...
01/11/2015 08:34:58:653 [05536] | INFO | Redirecting 'IDirect3DDevice8::CreatePixelShader(00F3F510, 0654A208, 06548D2C)' ...
01/11/2015 08:34:58:653 [05536] | INFO | > Disassembling shader and translating assembly to Direct3D 9 compatible code ...
01/11/2015 08:34:59:153 [05536] | INFO | Loading effect from "\POTC4\ReShade.fx" ...
01/11/2015 08:34:59:466 [05536] | ERROR | Failed to compile effect on context 0110DF30:

\POTC4\Shader@0x013603C8(27,: warning X3557: loop only executes for 0 iteration(s), forcing loop to unroll
\POTC4\Shader@0x013603C8(23,104): error X3508: '_NSSFX_NSSMAASearchDiag1': output parameter 'e' not completely initialized


01/11/2015 08:34:59:553 [05536] | INFO | Installing delayed hooks for "C:\WINDOWS\system32\ws2_32.dll" (Just loaded via 'LoadLibraryW("msdmo.dll")') ...
01/11/2015 08:34:59:555 [05536] | INFO | > Found 8 match(es). Installing ...
01/11/2015 08:34:59:698 [05536] | INFO | > Installed 8 hook(s).
01/11/2015 08:35:01:087 [05536] | INFO | Redirecting 'IDirect3DDevice8::CreateImageSurface(00F3F510, 1920, 1080, 22, 07225A7C)' ...
01/11/2015 08:35:03:439 [05536] | INFO | Redirecting 'IDirect3DDevice8::CreateImageSurface(00F3F510, 1920, 1080, 22, 00D3FA24)' ...
01/11/2015 08:35:12:146 [05536] | INFO | Exiting ...
01/11/2015 08:35:12:146 [05536] | INFO | Uninstalling 28 hook(s) ...
01/11/2015 08:35:12:505 [05536] | INFO | Exited.


I have deleted Fraps and done my best to disable AMD CCC. It is set to "use application settings" and all presets are deleted. This has no affect on modern games but the old ones look terrible with no AA. Reshade does not add enough to make dots and dashes go away.

This might happen if SMAA_MAX_SEARCH_STEPS_DIAG was defined to zero. Check your SweetFX settings file, it has to be some higher (usually 8 or 16).

That was indeed the problem! All of those settings were at 0. I moved them to the center of the sliders and SMAA is now working with no errors.

This game necessarily has very long draw distances and it looks ok up close now but anything at any distance looks bad.

Default SMAA settings:

Well, the error messages are solved, but there is still no AA at all. I tried Reshade with a prototype game using Unity5 and there is no AA there either.

Post-processing AA can only do so much.

crosire wrote: Post-processing AA can only do so much.

SweetFX AA keeps giving me a Gamma bug?

Quentin-Tarantino wrote:

crosire wrote: Post-processing AA can only do so much.

SweetFX AA keeps giving me a Gamma bug?

In some games though I don't know what causes it and I think it was improved upon in a earlier ReShade release as well so it should hopefully be very rare, currently there's only one game I'm playing that has that issue in now and that's Victor Vran.