(Dynamic/Adaptive) 3d lut + white\black point

i was thinking to improve 3d lut shader, to allow different lut textures to apply in different cases, but without having actual data from game about whats going on there its kinda hard, but i have few ideas, not sure however how to implement them:

1) Set array size in config, size would be number of actively used lut textures. it could be 3 or could be 24
2) List all loot textures in order in array inside config
3) List average colors in another array with same size, order should be relative to lut textures from pre array
4) Start a loop, aproximate average input color, and apply lut tex texture according to average color detected
5) somehow do smooth crossfade between luts to avoid sudden color changes
6) take care about performance

Problem is that i dont know how i can do some of that in hlsl, for example if curent colors would not be perfect match to one from array, how to deal with averaging between input and desired value with good performance without using extra loops or recursion? How to smooth out fade between previous and new lut and? Which is the best way to sample average scene colors?

Well its could be easier to just tell the difference between well lit and dark areas and just apply 2 different luts for nights or dark indoors and days or bright outdoors, but for more complex scenario (for example in games like rage there is a different color grading in different locations, some has green tint, other purple and i want to use that as indicator of current location and apply lut per location like this) that would be way harder.

Any ideas, samples of code and so on?

P.S. - there is also idea to somehow built virtual histogram of colors and generate lut in real time based on input, that will, for example correct white and black point (however it would be good if UI areas would be excluded from sampling by some mask), but i have no idea how i can do that in pure hlsl.

If you could sum up in a few simple words, what is the goal of this request/idea?
What effect should be applied and why?

The scene brightness is already calculated for the ambient light shader and some other effects from the lighting domain.
Eye adaptation is therefore implemented but whats the purpose/relationship of various color changes depending on scene brightness?

LUT is currently a very cheap option for color correction. Of course you can hook it up on scene brightness detection, if you do that already anyway, but for LUT alone this is quiet a performance impact.

Smooth transitions between LUTs are costly cause they involve storing previous "frame" information. Smooth transition of scene light on the other hand is given by design.
-Conceptual you can store the current mix of LUTs in an addition texture that you update for each frame-

I do not quiet get the dynamic white and black point calculation. If the purpose is to compensate oversaturation then there would be no difference to the already existing adaptation algorithm that, as said above, takes scene brightness into account.

are u saying a 3d + time....I mean 3d + white level = 4d lut?
totally doable but require huge among of time to tweak from preset author's end to have a satisfying result.

The interpolation between LUT nodes are basically linear, even if using gaussian to smooth out the transition peak, you still need at least 8 or 16+ levels to properly defined the curve without creating artifacts, which means 8+ work per preset for authors. _(´ཀ`」 ∠)_
.
might as well try to tweak a proper adaptation / tonemapping curve.

@Ganossa
what do you meant by performance impact?

He seems to want to control which LUT is picked depending on the scene brightness. Calculating the scene brightness causes impact on performance. If you compare that with the usual impact of a LUT, this is rather heavy.

but if you compare to adaptation, the performance difference should be minimum.

Right, as I said, if its used already for ambient light and Co there is basically no impact. Just for LUT alone it is very likely impact that might not be worth the effect.

The germ of the idea is still sound, the problem with LUTs or any color modding is that in-game grading changes. Case and point would be 99% of the profiles for GTAV, the high contrast nights look beautiful but by day it's a horrid mess. Perhaps LUTs are not the way to go but having a color profile that would trigger based on scene lighting is definitely a step forward.

/2cents

I think it would technically be no problem to linearly move between 2 LUT profiles for "day" and "night". If you wish, I can implement it conceptually in the next few days.
We could have then the option to hook onto or trigger the scene brightness detection in case "night" LUT gets activated and having the "day" LUT as base LUT which would otherwise be handled just the way it is right now.

you are just complicate things up, a 4d lut will work.
currently 3d lut here has grid dimension (grid depth) x 1, you can expend it to (grid depth) x (white depth)
and the pixel dimension would be (grid depth * grid depth) x (white depth * grid depth)

ie, if someone wish to use grid depth 16 and white depth 4, the outcome palette would be 256 x 64 compose by 4 strips of 256 x 16 3d lut.

If your comment was to me kingeric, from a users standpoint, you cannot solve a 2 LUT problem as comfortable with 1 4D LUT (not talking about storage possibility here) because as a user you cannot edit a 4D LUT as you can edit 2 3D LUTs. If you simply want (to edit) two different LUTs for "day" and "night" as Mobeeuz said (we did not get OPs verification on the original idea yet), 2 separate LUTs is A ) the most straight forward (-> very uncomplicated) and B ) the most comfortable solution for the actual user of this feature.

EDIT: Once we want to have more differentiation (so more than the 2 LUTs) and of course from a sole dev point of view, I would definitely agree to store the same information in a 4D LUT, also to minimize used textures, though I still see it problematic to be edited by the user.

whats the problem?
simple as MS paint can combines 3d luts into 4d....there are even some online stitcher available if anyone got lazy.

ㅋㅋㅋ, that kind of confidence I might have lost in time

Anyway, you are right, it should be that simple.

Ganossa wrote: ㅋㅋㅋ, that kind of confidence I might have lost in time

Anyway, you are right, it should be that simple.

hm well would it be possible to just make a tonemapping adaptive based on user input so if you went into a bright area it would have specific settings for that based on how the user wanted on light input then if they went into a darker area it would have its on settings for this as well could be wrong but i dont think this would be using a texture at all and do what the user wanted in a way

tone map with numeric values and different params ends up doing same thing ad 3d lut texture, except that 3d lut texture are easy to visually edit in any editor, while find tuning numerit values without understand what exactly shader author controls by them - pain in ass.

In terms of performance 3d lut shader seems like very fast to me, just lookt at one from tunning pallete:

Problem - it is applied every time, no matter what, even in menu, ui, at different weather conditions a. In some games like GTAV it works fine (corrected redish dimmed colors with incorrect white and black points, took me dozens of screenshots to aproximate best values, did same for witcher but unfornutally there results was not that good, days looks really nice now, but nights are too bright) but in others like Witcher 3 smart adaptation is required with different lut textures for different cases.

Recently i digged into Fallout 4 files and found there hell lot of different luts applied to weathers, time of days and indoors, and transition between them is really smooth, i never notticed how one lut changes another (for example in games like GTAIV when you are indoors you have more saturated colors, once you leave door trigger activate and in second all colors turns to shades grey - that always annoyed me) so this gave me idea to make similar shader. which would sample current environment and select lut texture assigned to this sample.

Second idea was to make separate shader or extension of this one, that wiill dynamically correct white and black point by sampling brightest pixels (except for region masked for UI, because ui usually not affected by bad color grading and while scen may have no correct whites at all, UI may have and this may lead to bad resuls)) and darkest, and turning them into real white and black. Part of this idea was this - provide 2 modes, one is always real time, may work fine, may glitch like hell, another one - press some button to make snapshot of current scene to temp buffer, sample it for white and black points, and then use auto corrected temp lut for rest of frame until button pressed again. So you can just rotate camera, walk here and there in game and press one button for auto adaptation of current lut, until you find most optimal one. (not sure if reshade allows to dump any autogenerated color output, but that would be nice feature).

I usually do that kind of thing in image editors based on screenshot and by looking at histogram, but there is always same pattern so i believe it could be automated.

This would be nice to cure problems like this forums.cdprojektred.com/threads/36088-A-...ewfull=1#post1749014 ( i gave detailed explaination there, plz read)

So this shade supposed to turn crap with incorect colors like this:



into this

Unless your looking for some over exaggerate change in color grading, the mentioned under exposure problem is normally solved by tonemappers with adaptation/ auto exposure as Ganossa mentioned (which is identical to your second idea), whether the adaptation is achieve by functions or 1d lut, there is no need for extra 3d lut ops.

//adaptation
docs.unrealengine.com/latest/INT/Engine/...cExposure/index.html
//3dlut usage
docs.unrealengine.com/latest/INT/Engine/...orGrading/index.html

about numeric 3d lut, 256x16 texture along requires 4096 groups of float3. don't know if that is even possible here.

1d numeric is ok, but again that is under the category of auto exposure.

Sorry, but thats not what i try to achive, you goit it all wrong. Its not eye adaptation! Its adaptive color correction, to fix poor unrealstic color grading. And idea of one is to sample white and black points, and another is to make adaptive 3d lut selector for different sampled scenes.

ok, then the mentioned 4d lut will work...
I misinterpret where you said "day is fine but night is too bright" while multiple 3d lut is more like for change of tone.

Changed my tuning palette shader code to also support 4D LUTs based on the scene brightness value. Seems to work as expected.
The performance is not dependent on how many 3D LUTs you want to run in parallel.

The current version is available on github if anyone want to try it out already.

Example:

4D LUT with 6 brightness layers (3D LUTs based of SH8 LUT collection)


Changing scene brightness causing the above LUTs to be applied smoothly


By the way, if you want to be more accurate about scene average or black and white points, you need more calculation power and at least an extended version of my current detection algorithm.

Interesting LUT software

$200-$250
You think they will hate me if i write a basic matching for free?