New shader: cinematic depth of field

OtisInf wrote: btw:

Marty McFly wrote: Hey would you mind if I "borrow" that physical based focusing code at some point?

To clarify: you may, as it's licensed under the BSD2 license, but you have to obey that license when borrowing that code, s copy the whole license in your shader file as instructed. A simple "some code by Frans Bouma" won't cut it.

In this case I'll do it myself, as I can only uphold my CC BY-NC-ND 3.0 if the entire code is mine. But I have other projects on my mind (smart DoF that can produce arbitrary sized blur discs at somewhat constant cost) so this'll be a problem for another day.

@crosire: I was under the impression that branches can be used if the code segments spared are very large and a lot of neighbouring pixels use it, so f.e. a checkerboard would be superbad but large areas like half of screen or so benefit a lot.

Yes, like I said, if all threads in a warp execute the same branch target, then that's fine. But divergent branches are bad.

crosire wrote:

OtisInf wrote: Tweaking things a bit, I noticed having a 'continue' is terrible for performance. I expected the compiler to emit fast code with jumps but apparently that's not the case!

You are writing code for the GPU, branches are the single worst thing you can do (anything that splits the code path, so that's ifs, fors, whiles, ... unless they can be unrolled which the compiler will try to do as best as possible). With the massive parallelism the GPU does multiple threads are always executed in a group (on NVIDIA this is called a warp of 32 threads, on AMD I believe it's called a wavefront of 64 threads). These run instructions in lockstep, meaning they always run the same instruction at the same time. As soon as you branch and one thread jumps to a different target than another in the same group, that thread has to wait for the first thread to finish before it continues, and then the first thread has to wait for that thread to finish going through its instructions until all threads converge to the same location again (after the if/else). If you are very unlucky and a single thread in a warp branches to a different location than all other threads, this effectivly means you force 31 threads to do absolutly nothing. That kills performance. So, it's important to remember that GPUs work very differently from CPUs (which are optimized for very fast branching via a lot of horse power put into branch prediction).

Thanks for the clarification I didn't know about the groups/warps/wavefronts aspects and it makes a lot of sense now

lol omg, just getting into this. Can't believe this is a thing. It's really amazing so far, just for the fact that I can finally control foreground DoF without destroying the background DoF experience. Amazing. Thanks! Pics soon, once I get a handle on what everything does. +10 kudos to you.

*some time later* Hmm, no matter what I do, I can't seem to achieve big bokeh circles. I can barely get the circles at all. The highlight bias seems to cause the screen to either be unaffected to blinding-white if the effect threshold is low enough for the effect to proc. Any tips on what I may be doing wrong? Been tooling with all the settings. Thanks so much!

Edit: Thought I should clarity that I'm soley looking for autofocus values/use.



That's about the strongest/biggest I can get the bokeh circles.

The game is No Man's Sky, btw, which is opengl, if that matters. Changed the settings more, but still this is the most I can get.



Ok think i figured out a little more. Looks gorgeous! But imo it could really benefit from a big decrease to the focus time (like perhaps at .2 seconds) to be gameplay-friendly. It just flickers way too much, like the other DoFs that come packaged with ReShade.




Static DoF is very nice with this though! Damn.

Cinematic DOF doesn't use any focus time, so it should be instantaneous (there's no code to blend between two focus points in a given time like in some other dofs, it simply focuses on the pixel at the focus point you've set)

To get really big highlights, the bokeh (the blur) circles at that point have to be big as well, so you have to have massive blur. Large lens/small aperture and large blur range gives that. If you have small(er) blur then highlights can't get bigger, as they are as big as the blur circle at the point So if the blur circle at the point is e.g. 10 pixels, then the highlight is 10 pixels.

i think you should consider adding focus time. our eyes don't instantly snap to focus, and neither do cameras. it would make the effect a lot more natural looking.

moriz1 wrote: i think you should consider adding focus time. our eyes don't instantly snap to focus, and neither do cameras. it would make the effect a lot more natural looking.

Perhaps I'll add it in the future. The main goal for the shader however was for screenshots, so with a static scene and for those focus time is really an annoyance. But I'll consider adding a panning between focus points in the future

This is fantastic!

Hmmm... Somehow the autofocus doesn't work in d3d9.
Using the manual focus plane works fine, but when enabling the autofocus the image just gets blurred evenly.
In the focus overlay the cross hair still shows up, but the other infos (focus plane, out-of-focus plane) do not.

Hmm. Don't have a dx9 game installed at the moment, will see if I can test some tomorrow. (and also if it still works in dx11!)

(edit)
Well it still works in DX11. It should then thus also work in DX9 if you use reshade 3.4.1, as 3.4.0 or lower has an issue in dx9 and a workaround is added to the shader for that, and I think you run into an issue in that workaround. I'll see with reshade 3.4.1 and a dx9 game whether it still works, there and with 3.4.0 to see if the workaround isn't broken.

(edit)
In dx9, using 3.4.1 it doesn't work indeed. Autofocus is broken. No idea why, as it should use the same code as on dx11. Looking into it.

(edit) I use darksiders 1 but everything's fucked in the edit controls in reshade 3.4.1 in this game, perhaps it's the high fps (95+) but I do have vsync on, which appears to be broken. I'll try remember me, which I know works, and if I can't find it, it's too bad...

Fixed: github.com/crosire/reshade-shaders/pull/102

Hi! So I'm absolutely in love with this shader, I think it's nicest DoF implementation I've seen yet. The others I've tried struggle a lot with foreground blur and edges in general, but the little trick you do to 'feather' edges fixes that and it's just gorgeous.

I'd like to bring up something about highlight management though, I believe it's the same thing that AssassinsDecree was talking about in the first part of his post. It's very hard to achieve pleasing highlight gain without overexposing areas of the image. For example, bokeh in night scenes looks dull and drab when in reality they should pop out, layering bright circles over the rest of the scene. The best illustration of this would be any starry sky in a game, as shown in the screenshots below. I understand that this likely has to do with games being SDR/LDR in nature and bokeh really only working magic in HDR environments. However, and I get that it's basically heresy to compare the two, I've seen ENB manage to do accurate highlight gain in SDR games.



Of course, you can see some pretty glaring flaws in the ENB implementation, most notably the weird green halo that forms around the character's head, then some depth estimation issues. But those stars! The ReShade one just looks sad in comparison.

Highlight gain's probably there for this specific reason, but however pushing the gain just causes other parts of the scene to get blown out, even with the threshold finetuned as precisely as possible (sorry for the lack of screenshot! DD:DA's tricky when it comes to taking them). Meanwhile, the ENB shader merely has "Bokeh Highlight" and it just does the magic with any bright, glowing parts of a scene, no threshold necessary.

You can find the shader here: enbseries.enbdev.com/forum/viewtopic.php?f=7&t=3224
Maybe they're doing it by targeting objects with emissive textures, since ENB hooks far deeper into a game than ReShade, but that's a hypothesis going off zero clue. I tried to read the .fxs and compare them, but unfortunately I don't understand a single thing. So I'm wondering if maybe you or others will, and whether that black magic can be transferred over to this black magic.

The shader also does have a few little tricks that could be very cool, like DoF noise as well as swirling bokeh & anamorphic distortion, just some additional effects to better seal this fantastic shader's cinematic appeal.

Thank you for all your great work!

as an extension for anyone who knows, what's the difference between an ENB shader and a ReShade shader? Could one be ported to the other, vice versa? My pedestrian understanding is that they're both .fx files, is all.

Thanks

About the highlights, there are several ways to do it, and e.g. marty's way (which is the same as kingeric's if I'm not mistaken) is basically finding the maximum and use that, which effectively comes down to a lot of highlights (every pixel that's a bit brighter than the surrounding pixels gets picked up as the maximum and extrapolated to all the CoC blur circles which touch it, hence low energy pixels still shine through as a highlight). I do this a little differently, where I crank up the blur average using a formula which leads to better highlights overall but has the downside that low-energy pixels which are a tiny bit brighter than the surroundings aren't picked up as a highlight and won't cause a large bokeh highlight.

However if you look at real-life photos with bokeh highlights, it works like what my shader does: only bright lights are amplified, like a streetlight or a light on a wall in the farplane. Tiny differences in brightness of a pixel aren't picked up, if you use the threshold a bit.

Your screenshot shows a sky with stars which aren't really that bright, and therefore aren't causing bokeh highlights even if you crank them up a lot. If you e.g. look at my shot from Shadow of the Tomb Raider:



you see it does work properly as the lights in the background are lamps hanging on wires, and as it is dark you get proper bokeh highlights from them. If I'd used the alternative here, I'd have gotten a tremendous amount of bokeh highlights, namely everywhere but that's not what looks natural (at least in my opinion )

I did have an implementation like marty and eric used before but it led to too much highlights which looked unnatural. It does have downsides indeed, if you have a bright element there which is brighter than the lights you want to turn into bokeh highlights, that bright element will ruin the shot. However in real life that's true as well. Same for stars: star lights aren't very energy heavy, taking a shot at night with stars in the far plane blurred, won't give you highlights at all. A bright lamp however will give you that.

Hope this helps

Ah, thanks for the response! I should have considered that stars are supposed to be pretty energy-weak, since they're pretty far away, my bad. However, I still get similarly weak bokeh in other lamp-lit scenes.


Here's some comparisons that ought to highlight what I mean, the top row is the 'reference' no DoF set, middle is ENB's DoF, bottom is yours. I attempted to match the two shaders as closely as possible, getting their bokeh circles to about the same size as I could as well as matching their luminosity (see the amber cluster in the first column).

The first two columns are of lamps in a scene. ENB correctly identifies them as light sources, and pushes their bokeh very bright. It stands out like an out-of-focus lamp should. Your DoF, however, manages very poorly, especially in the first column. Surrounded by reflected light, it can only manage a weak circle where the lamp should be. The second column's lamp fairs better since it's in higher contrast. The ENB one definitely isn't perfect either, you can tell it's bokeh-ing all the reflected light as well, however the lamp is correctly exaggerated.

The third column is of a tarp that's been lit by the lamp in the second column. This is more to demonstrate the highlight gain and threshold differences. I wanted to push your DoF's detected light sources to similar luminosity as that in the ENB, however you can see that in the last row of the third column the tarp becomes horrendously overexposed before even getting to a resembling brightness.

The last column is just two goblins patrolling in the night. More of a high-energy light source example, and really just there for additional comparisons' sake. The ENB shader identifies more things bokeh compared to your shader, however it could be argued that really only the torches should have been bokeh'd rather than all those extra details.
additionally, you might wanna note that you can see a silhouette of the goblins through your shader, shouldn't be too much of an issue and rather more extraordinary, but I thought I'd just point that out.

Your implementation is definitely more pleasing than the ENB one, colours are more accurate to in-game and it doesn't look as 'posterised' as the ENB one does. I'm just wondering how ENB manages to identify light sources correctly without having to overexpose other parts of the scene.

I'd assume it works really nicely in SOTTR since that's a new title and games nowadays tend to have proper dynamic range (i.e. identified light sources can register above a luminosity of 1 and the light they cast is clamped below 1, allowing depth of field implementations to create proper bokeh), but I like using ReShade to pretty up older/less demanding titles that may not have the same level of dynamic range management. I could be wrong though, I don't own SOTTR (nor do I have the machine to run it), so I'm just going off what I know about depth of field and how it works in 3DCG.

I guess a suggestion I have would be to implement some sort of levels for preprocessing? I'm assuming threshold just clamps everything under a certain luminosity down, so do forgive and correct me if I'm wrong. But I'm wondering if increasing contrast of what to defocus with the fine controls of B/W levels would help with this highlight query I have.

Full sizes of the screenshots used:

How high did you crank up the highlight gain? As that's pretty important It can be cranked up pretty high, overbrighting everything a lot, so it's important to know that value before I can answer your post, thanks

Oh, I had it cranked to 30, any higher and the highlights would begin to become unnaturally bright. Threshold was at 0.4, anything above would begin to cull the effect too much (especially for the lamp in the first column, pushed any higher and its bokeh circle just disappeared).

I would argue that the Cinematic DOF looks much more realistic than the ENB one judging by those screenshots. The ENB DOF puts bokeh all over the place. In other words it does not correctly identify light sources, insted it cranks the bokeh values so high that every light source got bokeh along with everything else too. For example, you can never light up dark canvas so much that it would actually produce bokeh but that just happens with ENB DOF by lighting it up wit a mere candle light.

The bokeh does not work like that every lightsource has it. The bokeh is caused by the contrast of the bright spot and its immediate surroundings. The bokeh in those lamp shots correctly gets blended with the light surroundings with Cinematic DOF. On the other hand, the fireplace has bokeh due to the strong local contrast.

To argue whether something is realistic, you must first understand the mechanics behind it. But that's about realism and I understand that it's not everyone's cup of tea. I too like unrealistically stong bokeh sometimes and it's fine. As far as I know, the realism is what Otis is after, but as always, it's up to him to decide.

altokitty wrote: Oh, I had it cranked to 30, any higher and the highlights would begin to become unnaturally bright. Threshold was at 0.4, anything above would begin to cull the effect too much (especially for the lamp in the first column, pushed any higher and its bokeh circle just disappeared).

30 is nothing I have the threshold usually around 0.6-0.7 and highlight gain very high too, perhaps even at 300-400. With the lights on the right, from the flames, if you want them to become brighter, you really have to increase the gain more and the threshold a bit lower perhaps (just a tiny bit, use alt-click to drag for small increase/decrease).

I'll go over your shots now First thanks for the feedback and effort you put into this Any feedback is much appreciated For clarity, I use an LDR monitor and reshade works in LDR space (as the framebuffer is LDR in a non-HDR output environment, so colors are [0, 1.0]. So my screenshot is taken from LDR pixels using LDR logic (color clips between 0, 1)

The middle row, left shot is an example why I reimplemented the highlight code I had (which was similar to marty's and eric's): you see a line of highlights and there are highlights basically everywhere. This doesn't make any sense. The bottom shot using my shader picks the light in the background as the most bright, and looking at the clear shot above I think that's also what is the case: the lamp looks white/yellow but the amount of pixels having that color is small, so they get blurred away but aren't staying that bright. I agree, if the bright spot in the bottomright wasn't there, you could crank up the gain way more and have a proper bokeh shape on the light. Doing so now will brighten the area in the bottomright as well, but that's in real life the same thing: that bright spot will overbright in your shot as well if you leave exposure open a long time: it's a bright spot, same as the lamp (as the lamp isn't giving out more lumen than the background in an LDR framebuffer)

2nd column, it's better, I'd crank up the gain here more and the threshold perhaps a bit as well to get rid of the other bright spot which is less bright. Also, look into increasing the # of rings to e.g. 6 and lower the usage of the post blur step (normally one should keep that below 0.2 to avoid washed out bokeh).

3rd column is a prime example why the technique used by the other shaders simply doesn't work, it's a painting with bokeh dots. As it's a large area which is brighter than the rest, you can't expect it to produce round bokeh shapes, as that's simply not how the artifact occurs in real life

4th column, here you clearly should have cranked up the gain more. It goes from 0 to 1000, so 30 is a very low gain. There are 2 bright spots, the flames, so you should get 2 bokeh highlights. Crank up the gain!

Keep in mind that a single pixel which is bright is blurred away in strength the more blur you use. It's not as in the other shaders where you get the same brightness of the pixel no matter how big your blur is. This is because otherwise you get results like the 1st and 3rd column of the second row.

Looking at the 4th column bottom picture you do have lots of undersampling (low number of rings, large blur range), which also doesn't help (as you have holes between the pixels). Try using 5 or 6 rings (if you aren't already) at least.

(edit) What Tojkar says, it's exactly that .

Thanks for the responses, guys! Heh, I thought maybe I had a decent understanding of how depth of field works, since I toy around with photography and 3D rendering here and there. But it's good to be get things wrong and learn every once in a while. I just thought, hey, there are light sources around in a scene, why don't they look like those fancy lil circles I see in photos sometime?

And yes, to Tojkar, I definitely do agree that Cinematic DoF is much more pleasing and accurate than ENB. I believe I stressed it quite often too, heh. I quite disliked how it did defocusing in general, especially the odd "painting" effect and the weird haloing that occurs, and then this shader popped up! I was honestly striving for some guise of realism, but it's hard to compare fantasy screenshots to anything in the real world, isn't it? Though, the point of my posts here were honestly more of just wondering why I can't get light sources to respond the way I want them to, which is strong contrasted bokeh in low-light scenes. It's understandable that maybe what I want isn't realistic, guess real life finds ways to get more disappointing every day.

30 is nothing I have the threshold usually around 0.6-0.7 and highlight gain very high too, perhaps even at 300-400. With the lights on the right, from the flames, if you want them to become brighter, you really have to increase the gain more and the threshold a bit lower perhaps (just a tiny bit, use alt-click to drag for small increase/decrease).

Woah, I see! I didn't know you normally cranked it up that high! The defaults are 0, 0, so, y'know, one might think eh, let's not push things too much. And thanks for the explanation on how things work. I don't really understand how ReShade works myself, I just enjoy what it does, so it's good to know that it operates uniformly across games without all the dynamic range stuff.

I appreciate you going over all the cases I highlighted, it's a very in-depth explanation. For the most part, I kept the settings at their default, so blur steps at 5, no post-blur smoothing, so on. I know you have to tweak some of those, but I kept them at default since this was mostly exploring how highlights worked. DD:DA (whoops, don't own Skyrim) was just the first game I thought of that supported ReShade and had ENB built by Boris himself for it, so I went to it after a few months of not touching the game.

The settings were kept uniform across the screenshots, because while I like to think about being a perfectionist, spending 10 minutes per screenshot just tweaking numbers is a bit excessive, no? (that's not even counting the time taken to properly frame and get the scene looking right!) Heh, I'd just like to have one uniform setting that I can always rely on no matter the scene. I guess a day and night set would be fine, since I imagine the sky gets ridiculously clipped with any semblance of highlight gain (it occurs with other techniques and bloom pretty often already, but I guess I should be able to correctly manage them now with this proper info).

Once again, thank you for going over this with me. I may honestly have thought that you somehow skipped out on a proper highlight management system, since the rest of your shader performs honestly just gorgeous! I'd argue it's even better than Marty's!
Perhaps on the next release, you could package it with proper preset values? Highlight gain and all already properly managed, so future users (and idiots like me) don't spend time writing essays figuring out how things work.

Well, I'll close off this conversation with some suggestions I'd love to see implemented in the future. Already said in a previous post, but I think swirling bokeh (how bokeh deforms towards the edges), anamorphic distortion, chromatic aberration and depth of field noise dithering would really help to seal this wonderful cinematic shader. Polygonal bokeh would be really nice too, gotta love me some fancy floating pentagons, y'know? I'll head off to tweak my settings more, maybe I'll come back with some pretty screenshots of my own. Keep up the hard work!

And I like lolis!