Reshade Question About Texture Flattening

6 months 1 week ago #1 by ModKujo
I've been playing around with various Reshade presets for the last year or so and I have this concept in my mind of a comic book look with flat colors, no texturing and minimal details. If you Google enough there are a few requests for this type of feature on various forums and YouTube comments, so I'm certainly not the only one thinking about this.

In games like Skyrim with a huge modding scene, there are many that have tried to get close (not all are Reshade): Impression ENB - An Aesthetic Cartoon Overhaul, SkyTOON, ASO - Artistic Skyrim Overhaul, Reshade cartoon.fx, Reshade paint.fx, and Wildrim. These are just the ones I can think of off the top of my head. Wildrim and ASO are the closest, but they are complete texture overhauls.

I get that Reshade is geared towards realism to a point. In many tools such as Photoshop to IG; they work on the principle of reading the pixels around them and then changing that pixel based on presets defined by the user and code. There are even dumb "got'cha" apps that take a picture and turn it into some sort of toon'ish thing. However, like most digital images, these shaders are sampling the pixels, doing the math, and producing an output pixel. Someone here is going to school me on that description... but I'd welcome a deeper explanation!

In my head I can see the code flattening out texture to some degree. Then where those colors have the greatest shift from one pixel to the next, apply a black line. Reshades cartoon.fx does the black line pretty well, and others have achieved the line as well, but those seem to be based on DOF. It's the color sampling and flattening that I don't understand.

I feel that this might be a limitation of game engines, but I don't yet understand how or why.

Could someone please explain to me why is it so hard to create a flat/cell shaded-ish/cartoon shader?

Thanks in advance!

Please Log in or Create an account to join the conversation.

5 days 23 hours ago - 5 days 23 hours ago #2 by Daemonjax
Replied by Daemonjax on topic Reshade Question About Texture Flattening
If I were to write my own cell shader effect, I'd want full depth buffer access, because that gives you the outlines of objects. Run the Displaydepth shader (or whatever it's actually called) to see the depth buffer so you can see how easy it would be to pull edge information from it.

I haven't looked at the cartoon shader, but I imagine it works that way.

Otherwise you'd have to do edge detection based on color, which won't look good imo.  Run the SMAA shader in color or luma mode and turn on the debug mode to view the detected edges -- see how it detects a lot of edges that aren't really edges you'd want to cell shade?  And that's a professional-grade shader where the majority of its purpose is to detect edges without depth buffer info.

Anyways, borderlands looks like borderlands because they created textures to specifically look good with their cell shader effect.  If you put different textures into that game (hires textures, for example) it wouldn't look right.
Last edit: 5 days 23 hours ago by Daemonjax.

Please Log in or Create an account to join the conversation.