replace shader during gameplay

Faisal.Roc
Topic Author

2 years 5 months ago #1 by Faisal.Roc replace shader during gameplay was created by Faisal.Roc

I'm making a new addon that replaces a shader on key press instead of at startup, I tried modifying the <example replace addon> but any attempts outside of create_pipeline and init_pipeline failed big time. So is it possible?
Any help would be highly appreciated.

Please Log in or Create an account to join the conversation.

crosire

2 years 5 months ago #2 by crosire Replied by crosire on topic replace shader during gameplay

Pipeline objects can't be modified once they have been created. Your only option would be to create your own pipeline objects with the shader code exchanged ("device::create_pipeline()") and then intercept when the game tries to bind the pipeline object specific to the shader you want to replace (in a "bind_pipeline" event callback, e.g. keep a list of the matching handles captured in the "init_pipeline" event callback and check if the bound handle is in that list) and instead bind your own ones (via "command_list::bind_pipeline()" and returning true in the "bind_pipeline" callback).

Please Log in or Create an account to join the conversation.

Faisal.Roc
Topic Author

2 years 5 months ago #3 by Faisal.Roc Replied by Faisal.Roc on topic replace shader during gameplay

How would I go to create pipeline objects in "create_pipeline"? or you mean pipeline_subobject instead?

what I did is set both:

shader_desc desc_ps_mod;
std::vector shader_code_ps_mod;

of my modified pixel shader in "create_pipeline"

and then

reshade::api::pipeline ps_handle;

struct __declspec(uuid("038B03AA-4C75-443B-A695-752D80797037")) CommandListDataContainer {
uint64_t ps_pipeline;
};

static void on_init_command_list(command_list* cmd_list) {
cmd_list->create_private_data<CommandListDataContainer>;();
}

static void on_bind_pipeline(command_list* cmd_list, pipeline_stage stages, pipeline pipeline) {
auto& state = cmd_list->get_private_data<CommandListDataContainer>;();

switch (stages)
{
//case pipeline_stage::vertex_shader:
//case pipeline_stage::hull_shader:
//case pipeline_stage::domain_shader:
//case pipeline_stage::geometry_shader:
case pipeline_stage::pixel_shader:
//case pipeline_stage::compute_shader:
state.ps_pipeline = pipeline.handle;
break;
}

}

Please Log in or Create an account to join the conversation.

crosire

2 years 5 months ago #4 by crosire Replied by crosire on topic replace shader during gameplay

std::unordered_map<pipeline, pipeline> g_replace_pipeline_lookup;

static void on_init_pipeline(device *device, pipeline_layout layout, uint32_t subobject_count, const pipeline_subobject *subobjects, pipeline original_pipeline)
{
    bool modify = false;
    shader_desc modified_ps = {};
    std::vector<pipeline_subobject> modified_subobjects(subobjects, subobjects + subobject_count);

    for (uint32_t i = 0; i < subobject_count; ++i) {
        if (subobjects[i].type == pipeline_subobject_type::pixel_shader && hash(static_cast<const shader_desc *>(subobjects[i].data) == PS_HASH_TO_REPLACE) {
            modify = true;
            modified_ps.code = REPLACED_PS_CODE;
            modified_ps.code_size = REPLACED_PS_CODE_SIZE;
            modified_subobjects[i].data = &modified_ps;
        }
    }

    if (replace) {
        pipeline modified_pipeline;
        if (device->create_pipeline(layout, subobject_count, modified_subobjects.data(), &modified_pipeline)) {
            g_replace_pipeline_lookup[original_pipeline] = modified_pipeline;
        }
        else {
            // Error
        }
    }
}

static void on_bind_pipeline(command_list *cmd_list, pipeline_stages stages, pipeline pipeline) {
    const auto it = g_replace_pipeline_lookup.find(pipeline);
    if (it != g_replaced_pipeline_lookup.end() && ...) {
        // Overwrite pipeline bound by the game with our modified one
        cmd_list->bind_pipeline(it->second);
    }
}

Please Log in or Create an account to join the conversation.

Faisal.Roc
Topic Author

2 years 5 months ago - 2 years 5 months ago #5 by Faisal.Roc Replied by Faisal.Roc on topic replace shader during gameplay

thanks for all the help, whoever still can't figure it out what's the role of cmd_list,
I also changed unordered_map by a vector because pipeline object doesn't have hash and key equality functions?

std::vector<std::pair<pipeline, pipeline>> g_replace_pipeline_lookup;

static void on_init_pipeline(device* device, pipeline_layout layout, uint32_t subobject_count, const pipeline_subobject* subobjects, pipeline pipelineHandle) {

    bool modify = false;

    shader_desc modified_ps = {};
    std::vector<pipeline_subobject> modified_subobjects(subobjects, subobjects + subobject_count);

    for (uint32_t i = 0; i < subobject_count; ++i) {
        if (subobjects[i].type == pipeline_subobject_type::pixel_shader && crc32(subobjects[i].data) == REPLACED_PS_HASH) {
            modify = true;
            modified_ps.code = REPLACED_PS_CODE;
            modified_ps.code_size = REPLACED_PS_CODE_SIZE;
            modified_subobjects[i].data = &modified_ps;
        }

    }

    if (modify)
    {
        pipeline modified_pipeline;

        bool found = false;
        for (auto& entry : g_replace_pipeline_lookup) {
            if (entry.first == pipelineHandle) {
                if (device->create_pipeline(layout, subobject_count, modified_subobjects.data(), &modified_pipeline)) {
                    entry.second = modified_pipeline;
                }
                found = true;
                break;
            }
        }
        if (!found) {
            if (device->create_pipeline(layout, subobject_count, modified_subobjects.data(), &modified_pipeline)) {
                g_replace_pipeline_lookup.emplace_back(pipelineHandle, modified_pipeline);
            }
        }

    }

}

struct __declspec(uuid("038B03AA-4C75-443B-A695-752D80797037")) CommandListDataContainer {
    uint64_t ps_pipeline;
};

static void on_init_command_list(command_list* cmd_list) {
    cmd_list->create_private_data<CommandListDataContainer>;();
}

static void on_bind_pipeline(command_list* cmd_list, pipeline_stage stages, pipeline pipelineHandle) {

    pipeline modified_pipeline;
    bool modify = false;
    for (const auto& entry : g_replace_pipeline_lookup) {
        if (entry.first == pipelineHandle) {
            modified_pipeline = entry.second;
            modify = true;
            break;
        }
    }
    if (cmd_list != nullptr && pipelineHandle.handle != 0 && isToggleKeyJustHit) {
        // Overwrite pipeline bound by the game with our modified one
        cmd_list->bind_pipeline(stages, modified_pipeline);
    }

}

Last edit: 2 years 5 months ago by Faisal.Roc.

Please Log in or Create an account to join the conversation.

crosire

2 years 5 months ago - 2 years 5 months ago #6 by crosire Replied by crosire on topic replace shader during gameplay

The command list is where the game records its rendering commands to, which is later send off to the GPU for execution. This is done every frame, so you have to overwrite the pipeline you want to replace every frame for the duration you need to, not just in one single frame. The "bind_pipeline" event calls your callback after the game binds a pipeline, which gives you the opportunity to change that binding by calling "bind_pipeline" again on the same command list.

The code you posted now overwrites every single pipeline the game binds, not just the one you specifically want to replace, which doesn't work (especially since in most cases it uses a uninitialized handle value, which will just crash).

You can use a unordered_map, just need to provide a hash function for the pipeline struct (e.g. like github.com/crosire/reshade/blob/07f601c9...eneric_depth.cpp#L53 ) or use the underlying uint64_t value as a key, rather than the handle struct.

Last edit: 2 years 5 months ago by crosire.

Please Log in or Create an account to join the conversation.

Faisal.Roc
Topic Author

2 years 5 months ago #7 by Faisal.Roc Replied by Faisal.Roc on topic replace shader during gameplay

Thank you for all the help finally got it working, just instantly crashes but for a moment it actually replaces it

std::unordered_map<uint64_t, pipeline> g_replace_pipeline_lookup;

static void on_init_pipeline(device* device, pipeline_layout layout, uint32_t subobject_count, const pipeline_subobject* subobjects, pipeline pipelineHandle) {
    s_data_to_delete.clear();



    std::vector<pipeline_subobject> modified_subobjects(subobjects, subobjects + subobject_count);

    for (uint32_t i = 0; i < subobject_count; ++i) {
        if (subobjects[i].type == pipeline_subobject_type::pixel_shader && crc32(subobjects[i].data) == PS_HASH_TO_REPLACE) {
            shader_desc modified_ps;
            modified_ps.code = REPLACED_PS.code;
            modified_ps.code_size = REPLACED_PS.code_size;
            modified_subobjects[i].data = &modified_ps;
            modify = true;
            reshade::log_message(reshade::log_level::error, "modify OK");
        }
    }

    if (modify) {
        pipeline modified_pipeline;

        if (device->create_pipeline(layout, subobject_count, modified_subobjects.data(), &modified_pipeline)) {
            g_replace_pipeline_lookup[pipelineHandle.handle] = modified_pipeline;
            reshade::log_message(reshade::log_level::error, "replace pipeline OK");
        }
        else {
            reshade::log_message(reshade::log_level::error, "Error modify pipeline");
        }

    }


}[/i][/i][/i]

struct __declspec(uuid("038B03AA-4C75-443B-A695-752D80797037")) CommandListDataContainer
{

};

static void on_init_command_list(command_list* cmd_list) {
    cmd_list->create_private_data<CommandListDataContainer>;();
}

static void on_bind_pipeline(command_list* cmd_list, pipeline_stage stages, pipeline pipelineHandle) {
    //if (cmd_list != nullptr && pipelineHandle.handle != 0)

    const auto it = g_replace_pipeline_lookup.find(pipelineHandle.handle);

    if (it != g_replace_pipeline_lookup.end() && isSwitchKeyPressed) {
        //Overwrite pipeline bound by the game with our modified one
        cmd_list->bind_pipeline(stages, it->second);

        reshade::log_message(reshade::log_level::error, "bind OK");
    }

}

Please Log in or Create an account to join the conversation.

crosire

2 years 5 months ago #8 by crosire Replied by crosire on topic replace shader during gameplay

Your creation code has a bug: The "modified_ps" variable is local inside the loop and is freed up as soon as the loop body is left, but you still have a pointer to it alive that is then attempted to be used to create the pipeline. That's accessing freed memory and will just create bogus results. I had allocated that variable outside the loop in the example pseudo-code intentionally so that this does not happen. There is also no need to create private data on the command-list, since it's not used.

Please Log in or Create an account to join the conversation.

Faisal.Roc
Topic Author

2 years 5 months ago #9 by Faisal.Roc Replied by Faisal.Roc on topic replace shader during gameplay

I completely removed "init_command_list" and still works but is it good practice? I mean shouldn't be initialized at least?

Please Log in or Create an account to join the conversation.

crosire

2 years 5 months ago #10 by crosire Replied by crosire on topic replace shader during gameplay

Why, what do you need to initialize?

Please Log in or Create an account to join the conversation.