Welcome, Guest.
Username: Password: Remember me

TOPIC: Shader/Preset Management Proposition

Shader/Preset Management Proposition 1 year 4 months ago #1

  • crosire
  • crosire's Avatar
  • Offline
  • Posts: 2782
  • Thank you received: 1584
I'd like to propose a concept on how shaders and presets could be managed in ReShade 3.0, which I think would be very benefitial for both users and developers. But I prefer to have some community feedback, before actually implementing it.

The idea is as follows:
The GitHub repository is used as main distributor for all shaders. But to keep it from becoming a huge blob of stuff like it is now, it should be split up into multiple Git branches. As an example: an "everything" branch, a "ssao-and-tonemapping" branch, etc., each designed for a specific task and only containing the files needed for that.
Now when booting a game ReShade was installed to (only the DLL, the main download shouldn't contain anything else), ReShade's in-game UI will pop up, allowing the user to select an existing preset or create a new one. If latter option is chosen, he/she can select one of the Git branches as the collection of shader he/she wants to use and configure. Those will then be downloaded from GitHub and loaded into the game. The user is allowed to change various variables those shaders expose and save the preset, which is essentially stored in a simple INI file containing the following data. Those preset files can then be easily shared, uploaded/downloaded and selected in ReShade.
  • Git branch, to specify which shaders are used
  • Git commit hash from when the download occurred (to make sure each user of the preset has the exact same set of files)
  • List of shaders and variables + values
  • Optional: Encoded texture data of stuff like LUT textures?
Also. Every shader should be self-contained. No overall framework. So it's easier to take one oút of context and put it somewhere else. And much, much easier for developers.

tl;dr
  • All shaders on GitHub
  • Multiple branches with different shader collections to prevent messy blob (as it is currently)
  • Presets stored in INI with branch name + commit hash + variable values
  • Download required shaders as specified in a preset from GitHub on-demand
Cheers, crosire =)
Last Edit: 1 year 4 months ago by crosire.
The administrator has disabled public write access.

Shader/Preset Management Proposition 1 year 4 months ago #2

  • OtisInf
  • OtisInf's Avatar
  • Offline
  • Posts: 146
  • Thank you received: 67
crosire wrote:
I'd like to propose a concept on how shaders and presets could be managed in ReShade 3.0, which I think would be very benefitial for both users and developers. But I prefer to have some community feedback, before actually implementing it.

The idea is as follows:
The GitHub repository is used as main distributor for all shaders. But to keep it from becoming a huge blob of stuff like it is now, it should be split up into multiple Git branches. As an example: an "everything" branch, a "ssao-and-tonemapping" branch, etc., each designed for a specific task and only containing the files needed for that.
There's a problem with that: duplication of files. A shader that's in multiple 'branches' is been duplicated, making maintaining that code harder. Branches are also a little cumbersome on Github's UI. Branches are used for making changes in isolation/a different path, and meant to ultimately being merged back to truck/master. Github doesn't support forking of branches, only repos. This can be a bit cumbersome too if you want changes to happen on a separate branch: the developer has to fork the complete repo.

I think what you're after is cherry-picked sets of files from the full set of shaders. Why not use something like:
- have all shaders in one set (the code), or perhaps we should call them 'effects', as I think that's how the general user sees them.
- have multiple files which define 'shader sets', e.g. 'all shaders', 'DoF based shaders', 'AA', 'SweetFX' etc. They're just simple files which define a set of shaders to load on startup, and the shaders to load are read from the full batch.

This does require configuration (the input / start parameters of a shader, so not which parameters there are, but what value they have) are separated and these are defined with the 'shader set' config.
Now when booting a game ReShade was installed to (only the DLL, the main download shouldn't contain anything else), ReShade's in-game UI will pop up, allowing the user to select an existing preset or create a new one. If latter option is chosen, he/she can select one of the Git branches as the collection of shader he/she wants to use and configure. Those will then be downloaded from GitHub and loaded into the game. The user is allowed to change various variables those shaders expose and save the preset, which is essentially stored in a simple INI file containing the following data. Those preset files can then be easily shared, uploaded/downloaded and selected in ReShade.
  • Git branch, to specify which shaders are used
  • Git commit hash from when the download occurred (to make sure each user of the preset has the exact same set of files)
  • List of shaders and variables + values
  • Optional: Encoded texture data of stuff like LUT textures?
Also. Every shader should be self-contained. No overall framework. So it's easier to take one oút of context and put it somewhere else. And much, much easier for developers.
Sounds good, except the branch idea is something that won't work I think. Imagine a bug in a shader which has to be patched: it's better to update the shader once than to have to merge downstream to branches which, as they now contain the code of the branched shader, run the risk of drifting apart. How I understand it, the things which differ per 'branch' are input / config values and which shaders are part of the set, not the shader code itself. Perhaps with some tooling for deployment can help

I know it's not an easy problem to solve... Relying on a master branch too has its downsides, e.g. the advantage of your proposed setup of just share a preset and it always works, is mitigated...

Perhaps you could instead store paths to shader files? E.g. github.com/crosire/reshade-shaders/blob/.../Otis/AdaptiveFog.fx for master version, github.com/crosire/reshade-shaders/blob/.../Otis/AdaptiveFog.fx for exact commit version? No branches needed, or perhaps you talk about the same thing, not sure ;) (there are also git 'tags' btw, where you can tag the master branch with versions, e.g. 3.1, 3.1.2 etc.) and refer to those as a 'releases version' in the file URLs for the shader code to download). This has the advantage that if a shader's code changes and is released in e.g. 3.2.0, a preset referring to e.g. tagged version 3.1.9 will still work.
Last Edit: 1 year 4 months ago by OtisInf.
The administrator has disabled public write access.

Shader/Preset Management Proposition 1 year 4 months ago #3

  • DeMondo
  • DeMondo's Avatar
  • Offline
  • Posts: 208
  • Thank you received: 174
@crosire:
Thank you for constantly improving ReShade.

Your plan to implement a ReShade ingame menu sounds & looks great & above all is clear.

This latest post from you, I have to admit it sounds confusing to me, but thats because Im totally non-techsavvy when it comes to GitHub repositories, shader-coding, etc.. the confusing part for me lays in yr: "it should be split up into multiple Git branches" & "All shaders on GitHub"
But all shaders are already on the GitHub, no?

yr: "Those preset files can then be easily shared, uploaded/downloaded and selected in ReShade."
Does this mean that sfx.thelazy.net/games/ will move more into the background or being replaced or?
Yr: "Also. Every shader should be self-contained." No clue what it means, is it comparable like the vid-container Matroska (.mkv files)?
That every part needed is inside the file?

For me as (super-n00b :) )-ReShade-user the most important thing is how its being installed, and the way its being edited. For me it would be great, if all different shader config values are placed into one big config file.
Thus SweetFX.cfg, McFX.cfg, GemFX.cfg, etc sitting in one file for example ReShadeMaster.cfg with clear separated sections for each shader effect.
Now when you want to alt-tab out the game & edit the ReShade settings on the fly, you have to search for the specific cfg file, thus it happens many times I do have 5 different cfg files open while during gameplay.
The administrator has disabled public write access.

Shader/Preset Management Proposition 1 year 4 months ago #4

  • crosire
  • crosire's Avatar
  • Offline
  • Posts: 2782
  • Thank you received: 1584
I see your point on using branches. I don't like using any more higher-level concepts like extra files specifying shader sets though. Too complex.

OtisInf wrote:
Perhaps you could instead store paths to shader files? E.g. github.com/crosire/reshade-shaders/blob/.../Otis/AdaptiveFog.fx for master version, github.com/crosire/reshade-shaders/blob/.../Otis/AdaptiveFog.fx for exact commit version?
It would be easy to download specific shader files alone this way. But any included files or other dependencies like textures wouldn't be resolved, so it doesn't work.

Fortunately, GitHub has a sophisticated REST API, which could be used to list the contents of the repository. If each shader was put in a separate subdirectory with all its dependencies in the same place, ReShade could then easily download those folders separately (via api.github.com/repos/crosire/reshade-shaders/contents/...).
Root
---- Effect 1
-------- Effect1.fx
-------- Texture.png
---- Effect 2
-------- Effect2.fx
...

An alternative idea: It would be much nicer to just specify a number of ZIP download paths in a preset from which ReShade gathers all required files. That way we can use GitHub for official stuff (using GitHub zipball), but also leave the option for preset creators to include custom shaders/files. We could even get rid of the main GitHub repository altogether then, so every developer can host his/her shaders on his/her own place of preference.

DeMondo wrote:
This latest post from you, I have to admit it sounds confusing to me, but thats because Im totally non-techsavvy when it comes to GitHub repositories, shader-coding, etc..
The post was mainly addressed at shader developers, so sorry for the technical details =).

DeMondo wrote:
yr: "Those preset files can then be easily shared, uploaded/downloaded and selected in ReShade."
Does this mean that sfx.thelazy.net/games/ will move more into the background or being replaced or?
The opposite actually. thelazy.net only allows sharing single plain text files, which would work greatly with the INI preset files I described.

DeMondo wrote:
Yr: "Also. Every shader should be self-contained." No clue what it means, is it comparable like the vid-container Matroska (.mkv files)?
That every part needed is inside the file?
Right now there is one file (ReShade.fx) loaded by ReShade which includes all other shader files. This makes it quite complicated to add new shaders, because you have to make sure you don't use the same variable names as others etc. (because they all end up in the same file).
ReShade 3.0 no longer loads one main file, but instead searches through various specified directories and loads all .fx files it finds separately. That's why each effect can now be in its own file without depending on anything else (Try to move one of the effect files from the 2.0 framework to a new location, it won't work, because depending on ReShade.fx and other stuff; With 3.0 this shouldn't be a problem anymore).
Cheers, crosire =)
The administrator has disabled public write access.
The following user(s) said Thank You: DeMondo

Shader/Preset Management Proposition 1 year 4 months ago #5

  • OtisInf
  • OtisInf's Avatar
  • Offline
  • Posts: 146
  • Thank you received: 67
crosire wrote:
I see your point on using branches. I don't like using any more higher-level concepts like extra files specifying shader sets though. Too complex.
agreed.
...
Fortunately, GitHub has a sophisticated REST API, which could be used to list the contents of the repository. If each shader was put in a separate subdirectory with all its dependencies in the same place, ReShade could then easily download those folders separately (via api.github.com/repos/crosire/reshade-shaders/contents/...).
Root
---- Effect 1
-------- Effect1.fx
-------- Texture.png
---- Effect 2
-------- Effect2.fx
...
I like this idea. The advantage is that you can still easily make patches through fork->fix->pull request. All effects are self-contained, in their own folder.
An alternative idea: It would be much nicer to just specify a number of ZIP download paths in a preset from which ReShade gathers all required files. That way we can use GitHub for official stuff (using GitHub zipball), but also leave the option for preset creators to include custom shaders/files. We could even get rid of the main GitHub repository altogether then, so every developer can host his/her shaders on his/her own place of preference.
Hmm. How are updates/fixes done in that case? Also keep in mind that some people (like yours truly) don't really like running code that's pulled from some random stranger's server: who knows what's coming with the file: the stranger has the control over what will downloaded at runtime (as s/he can change the files at any time and anyone using the shaders will get the changed files).

Also, if one server is down or not reachable, you can't use that effect. I like the github idea more.
The administrator has disabled public write access.

Shader/Preset Management Proposition 1 year 4 months ago #6

  • DeMondo
  • DeMondo's Avatar
  • Offline
  • Posts: 208
  • Thank you received: 174
@crosire: Thank you for yr detailed answer. One of the most interesting parts for me is in this what you wrote: ".. and loads all .fx files it finds separately."
So it means we can "stack" multiple .fx files.. that would be a huge plus for ReShade users, the ability to test out new shaders (via the fx files, like Marty's new MXAO) and combine them with the than existing 3.0

I have to say Im hyped for your upcoming ReShade 3.0 and when I look through the ReShade forums many others as well, ReShade was already a success but 3.0 I do believe will be a megahit. B)
The administrator has disabled public write access.

Shader/Preset Management Proposition 1 year 4 months ago #7

  • crosire
  • crosire's Avatar
  • Offline
  • Posts: 2782
  • Thank you received: 1584
Yet another idea would be to link to online fx files in a preset directly. And have the "source" annotation for textures accept web links as well (+ a download cache to speed up loading).
It solves the problem with arbitary files in ZIP archives and it doesn't require developers to use GitHub (not everybody working with ReShade does). But, any "#include"s would need to be specified somehow aswell. I'm not sure how to handle those best?

[http://reshade.me/shaders/Test.fx]
SomeCol = 1.0, 0.0, 0.0, 1.0

[https://github.com/crosire/reshade-shaders/ReShade/Shaders/SMAA.fx]
SomeVar1 = 10.0
SomeVar2 = 12.0, 3
Cheers, crosire =)
Last Edit: 1 year 4 months ago by crosire.
The administrator has disabled public write access.