Edge Smoothing

  • Posts: 206
1 year 3 months ago - 6 months 2 weeks ago #1 by Duran.te
Hi, everyone.
I was wondering if someone could make an edge softening shader, like a gaussian blur filter limited to the edges.

I'm not a fan of rough edges. it feels like smooth edges seems to give more cinematic look to the scene.
Not to mention all those games that uses too aggressive sharpening filters, causing ringing artifacts around the edges:



An example of ringing artifacts: Notice those white haloes around edges.

Here's a simulation of the effect I'd like to achieve, made with photoshop.

I applied gaussian blur to the whole image and then I made a mask using with the photoshop's "find edges" filter, so that the textures (especially the snow and the threes, in this case) are not affected by the blurriness.

Also, the subtle ringing artifact is gone.
Original Shot




Edge Smoothing




Edge Mask



I know that Reshade's FXAA filter could be a solution, but the problem is that it messes with already anti-aliased images:
(In-game TAA is on)

The following user(s) said Thank You: BeTa, acknowledge, Viper_Joe, Dom324

Please Log in or Create an account to join the conversation.

  • Posts: 273
1 year 3 months ago - 1 year 3 months ago #2 by brussell
Well, funnily I've just implemented a way to reduce halos in my sharpening shader. It's nothing sophisticated, but I'm quite satified with the results. Basically I'm just adding sharpness depending on pixel brightness, thus reducing edges in high-contrast areas (like skies).

Here's a comparison (left: original, middle: less sharpness for bright pixels, right: standard sharpness)


Of course due to the uneven sharpening some details can look slightly odd (text etc.), but imo it can achive better results than by just clamping the unsharp mask.
Maybe I will post the code here soon. Although it may lack context, here is the important line:
fine *= (1-saturate(pow(abs(color.xyz), 1 / (fSharpen_DynamicCurve + 0.001))));
The following user(s) said Thank You: BeTa, jas01, Viper_Joe, Duran.te

Please Log in or Create an account to join the conversation.

  • Posts: 159
1 year 3 months ago - 1 year 3 months ago #3 by Fu-Bama
Check out this topic reshade.me/forum/shader-presentation/395...n?limitstart=0#26156

Created sharpening filter that addresses just that issue using depth high-pass mask and it's already in default repository under FilmicAnamorphSharpen.fx



Depth pass is required for it to work, but you can also try using "Sharpen clamping" setting without depth pass to reduce halos.
If you're looking for more performance-wise version without depth masking, visit my Shaders repository (link under my profile information at the bootom of this post), look for FilmicSharpen.fx
The following user(s) said Thank You: Duran.te

Please Log in or Create an account to join the conversation.

  • Posts: 206
1 year 3 months ago #4 by Duran.te
This certainly solves the first problem. However, as I mentioned above, there's nothing to do with already - oversharpened images, in that case an edge softener shader would help a lot.

Please Log in or Create an account to join the conversation.

  • Posts: 206
10 months 2 weeks ago #5 by Duran.te
Bump.

Topic updated, added reference shots.
Could someone help this case?

Please Log in or Create an account to join the conversation.

  • Posts: 155
10 months 4 days ago - 10 months 4 days ago #6 by jas01
You can try using gaussain blur shader or NFAA.fx from this shader pack. Could work in a few cases. I'm unable to give you any better solution here.

Please Log in or Create an account to join the conversation.

  • Posts: 206
10 months 1 day ago #7 by Duran.te
Unfortunately, since NFAA is an antialiasing shader, it causes some artifacts with in-game antialiasing on (as you can see in the last screen I posted).
It would also be nice if values like intensity and radius could be controlled with dedicated sliders, rather than just the threshold one.
Thanks for help, anyway. :)
The following user(s) said Thank You: jas01

Please Log in or Create an account to join the conversation.

  • Posts: 142
10 months 21 hours ago - 10 months 20 hours ago #8 by Insomnia
A lot of modern games have a sharpening effect enabled. Try look for sharpening settings in the game's ini files.

Please Log in or Create an account to join the conversation.

  • Posts: 7
9 months 4 weeks ago #9 by Dom324
Your screenshots look truly great (except that blur and slightly altered colors on the backpack), I can imagine number of games where this could be used.

Please Log in or Create an account to join the conversation.

  • Posts: 3
9 months 3 weeks ago - 9 months 3 weeks ago #10 by S3r1ous
Combination of Texture LOD -0.250(driver controlled LOD is off), AMD Sharpening filter (i used a version ported to Reshade) and NFAA is working really well for me
oh and i use Deband to smooth out some of those banding problems, you can see it really well in the skies in some games

Please Log in or Create an account to join the conversation.

  • Posts: 206
8 months 2 weeks ago #11 by Duran.te
For anyone interested: Pirate_FXAA from PirateKitty's shaders collection works pretty well as an edge smoothing filter, since it seems to not causing artifact on thin lines, like texts: it just smooths them (Guess it's using an older FXAA version).

You'll find it on PirateKitty's GitHub repository: github.com/Heathen/Pirate-Shaders
The following user(s) said Thank You: jas01, Viper_Joe

Please Log in or Create an account to join the conversation.

  • Posts: 3
6 months 2 weeks ago - 6 months 2 weeks ago #12 by j4712
Just a little concept i made up tonight. might make a proper version with depth+color detection and some settings if you are interested :)
/**
 * Smooth Edges
 * by Jakob W. ~ j4712
 * made for Duran.te 
 * NOT FINISHED JUST A CONCEPT FOR NOT USING DEPTH FOR THIS TASK WITHOUT DESTROYING INGAME AA
 */
 
#include "ReShade.fxh"
#include "ReShadeUI.fxh"

texture TexColorBuffer : COLOR;

sampler SamplerColor
{
	Texture = TexColorBuffer;
	AddressU = MIRROR;
	AddressV = MIRROR;
};

texture blurredColor
{
	Width = BUFFER_WIDTH;
	Height = BUFFER_HEIGHT;
};

texture edgesTex
{
	Width = BUFFER_WIDTH;
	Height = BUFFER_HEIGHT;
};

sampler blurredSampler
{
	Texture = blurredColor;
};

sampler edgesSampler
{
	Texture = edgesTex;
};

float rgbLuma(float3 color)
{
	return float3( color.r * 0.2126,  color.g * 0.7152,  color.b * 0.0722);
}

float3 BlurringPass(float4 vpos : SV_Position, float2 texcoord : TexCoord) : SV_Target
{
	float kernel[9] = { 1, 2, 1,
						2, 4, 2,
						1, 2, 1 };
						
	float3 color = float3(0.f,0.f,0.f);

	int kernalSumm = 0;
	
	for (int y = 0; y < 3; y++)
	{
		for (int x = 0; x < 3; x++)
		{
			int index = y * 3 + x;
			
			float3 pixelColor = tex2D(ReShade::BackBuffer, texcoord + (float2(x-1, y-1) * ReShade::PixelSize)).rgb;
			color += pixelColor * kernel[index];
			kernalSumm += kernel[index];
		}
	}
	
	color /= kernalSumm;
	
	return color;
}

float3 DetectSobelPass(float4 vpos : SV_Position, float2 texcoord : TexCoord) : SV_Target
{

	float kernel_x[9] = { 	-1.f, 0.f, 1.f,
							-2.f, 0.f, 2.f,
							-1.f, 0.f, 1.f 	};
							
	float kernel_y[9] = { 	1.f, 2.f, 1.f,
							0.f, 0.f, 0.f,
							-1.f, -2.f, -1.f };

	float3 color = float3(0.f, 0.f, 0.f);
		
	float edge_x = 0.f;
	float edge_y = 0.f;
	float kernelSumm_x = -1.f;
	float kernelSumm_y = -1.f;
	
	for (int y = 0; y < 3; y++)
	{
		for (int x = 0; x < 3; x++)
		{
			int index = y * 3 + x;
			float brightness = rgbLuma(tex2D(blurredSampler, texcoord + (float2(x-1, y-1) * ReShade::PixelSize)).rgb);
								 
			edge_x +=  (brightness * kernel_x[index]);	
			edge_y +=  (brightness * kernel_y[index]);	
			
			kernelSumm_x += kernel_x[index];				
			kernelSumm_y += kernel_y[index];
		}			
	}
	
	edge_x = ((edge_x / kernelSumm_x) / 2.f);
	edge_y = ((edge_y / kernelSumm_y) / 2.f);
	
	color.r = edge_x;
	color.g = edge_y;
	
	
	
	return sqrt((color.r * color.r) + (color.g * color.g));
}

float3 EffectPass(float4 vpos : SV_Position, float2 texcoord : TexCoord) : SV_Target
{
	float3 oColor = tex2D(SamplerColor, texcoord).rgb;
	float3 bColor = tex2D(blurredSampler, texcoord).rgb;
	float  edgeStrength = 0;
	
	float kernel[9] = { 1, 2, 1,
						2, 4, 2,
						1, 2, 1 };

	int kernalSumm = 0;
	
	for (int y = 0; y < 3; y++)
	{
		for (int x = 0; x < 3; x++)
		{
			int index = y * 3 + x;
			
			edgeStrength += tex2D(edgesSampler, texcoord + (float2(x-1, y-1) * ReShade::PixelSize)).r * kernel[index];
			kernalSumm += kernel[index];
		}
	}
	
	edgeStrength /= kernalSumm;
	
	if (edgeStrength < 0.075)
		edgeStrength = 0;
	edgeStrength = saturate(edgeStrength * 5);
	
	return lerp(oColor, bColor, edgeStrength);
}



technique SoftEdges
{
	pass BlurPass
	{
		VertexShader = PostProcessVS;
		PixelShader = BlurringPass;
		RenderTarget = blurredColor;
	}
	
	pass DetectSobel 
	{
		VertexShader = PostProcessVS;
		PixelShader = DetectSobelPass;
		RenderTarget = edgesTex;
	}	
	
	pass Effect 
	{
		VertexShader = PostProcessVS;
		PixelShader = EffectPass;	
	}
}
i know its messy and that there is a lot of code that cut be cut short. i didnt do any optimization
The following user(s) said Thank You: Viper_Joe, BFG9000, Duran.te

Please Log in or Create an account to join the conversation.

  • Posts: 206
6 months 2 weeks ago #13 by Duran.te
You're kiddin? It looks so neat!
I'd almost given up hope of seeing a custom shader made just for this.
A depth+color detection version with adjustable settings for stuff like threshold, radius and intensity would be so awesome, but I don't want to pressure you about that.

Thanks a lot man, you rock ;)
The following user(s) said Thank You: j4712

Please Log in or Create an account to join the conversation.

  • Posts: 120
3 months 1 week ago #14 by prod80
My sharpening shader has options for strength, depth masking, and a limiter for highlights (which tend to oversharpen). Perhaps worth a shot.
github.com/prod80/prod80-ReShade-Repository
The following user(s) said Thank You: Duran.te

Please Log in or Create an account to join the conversation.