Actually I was about to write this big thing explaining what Ceejay.DK said about prediction and edge detection but I actually just found it buried in the SweetFX reddit
Here is direct copy and paste:
CeeJayDKSweetFX Developer[M] 1 point 4 months ago
You are correct Color edge detection is slightly better than Luma Edge detection (which only detects edges from a monochrome perspective).
Luma edge detection is a tiny bit faster on Nvidia cards but on AMD cards the color edge detection is actually the faster one.
I suggest Color edge detection which is also the default.
In SweetFX 2.0 depth edge detection is also supported in games that reshade provides a working depth buffer view for.
Depth buffer detection only picks up on edges to objects and not any edges in foliage (or other alpha textures) or in textures - which can be both good and bad depending on the game and your preference. But it has no or almost no false positives (where SMAA thinks there is an edge when there really isn't) and it does not process the UI, so text is unharmed.
And then there is the predication mode which is a hybrid of depth edge detection and color edge detection - it uses both to find edges, to ensure it caught (almost) all the edges.
Here is the actual thread:
www.reddit.com/r/sweetfx/comments/2yfhw7/edge_detection/
Hopefully this helps you out a little:
And about the Threshold. Basically there is a point of diminishing returns for all those values. The Stock ones have been deemed good enough for most applications but you can increase all the values if are noticing missed spots. I would definitely use prediction first though if you are noticing missed lines and the like but you can increase the values too.
I use:
#define SMAA_THRESHOLD 0.08 //[0.05:0.20] //-Edge detection threshold. If SMAA misses some edges try lowering this slightly. I prefer between 0.08 and 0.12.
#define SMAA_MAX_SEARCH_STEPS 25 //[0:98] //-Determines the radius SMAA will search for aliased edges
#define SMAA_MAX_SEARCH_STEPS_DIAG 16 //[0:16] //-Determines the radius SMAA will search for diagonal aliased edges
#define SMAA_CORNER_ROUNDING 25 //[0:100] //-Determines the percent of antialiasing to apply to corners. 0 seems to affect fine text the least so it's the default.
Basically I put some AA on corners and increased the other values slightly. It effects text more but it's a non-issue in the game I applied it too.
I used to go all out and use the Ultra-settings which were like 0.05 threshhold max all other values (50 corner rounding) but I no longer do that because there is a point where it no longer makes a difference really.
And people don't use prediction in presets because:
1: they don't know what it does.
2: they see no improvement.
3: It doesn't work online due to no depth-buffer access.
4. (I think!) Sweetfx+Reshade cannot access Log-depth still so It has no access(and thus no effect) in games like GTA V and The Witcher 3.