Vulkan layer: replacing the present's wait semaphores hung the GPU with DXVK 3.x
- ShadowCat
- Topic Author
Less
More
17 hours 39 minutes ago #1
by ShadowCat
Vulkan layer: replacing the present's wait semaphores hung the GPU with DXVK 3.x was created by ShadowCat
Summary
I maintain a small ReShade-compatible Vulkan layer for Linux (it runs ReShade FX effects through a Vulkan layer instead of through Wine). In vkQueuePresentKHR it did what ReShade's Vulkan hook does: it submits the effect work waiting on the present's wait semaphores and then replaces the present's wait list with the semaphore the effect submit signals.
That worked with every application I tried until I ran a game through DXVK 3.1.1-97 (Proton Experimental). It creates its swapchain with VK_KHR_present_id2 / present_wait2 flags and attaches VkSwapchainPresentFenceInfoEXT and VkSwapchainPresentModeInfoEXT to every present (VK_EXT_swapchain_maintenance1). With a heavy effect chain the GPU hung within a minute or two (RTX 3060, driver 615.78.08: Xid 69 "Class Error", once Xid 109 "CTX SWITCH TIMEOUT"). The same game and effects with GE-Proton 11-7 (DXVK 3.1-19, which does not use those extensions) never hung, and a light preset on the same DXVK did not hang either.
What fixed it
Not replacing the wait list. The effect submit waits on the application's semaphores and signals the same semaphores again; the present then waits on exactly what the application handed over, and the rest of the present info (including the pNext chain with the present fence / present id) is untouched. Validation layers stay clean. With that change the same game, same Proton, same heavy preset ran without any Xid (it hung every time before).
Why it may matter for ReShade
source/vulkan/vulkan_hooks_swapchain.cpp, at the end of vkQueuePresentKHR:(and the immediate command list flush points wait_semaphore_info at the semaphore it signaled). So the present stops waiting on the semaphores the application passed. For an application that relies on VkSwapchainPresentFenceInfoEXT ("signaled when the semaphores waited on by this present can be reused") or on present ids, those are no longer the semaphores the presentation engine waits on. I do not know which part triggers the hang on NVIDIA; I only know that not replacing them removes it. The same function already has a branch for RTX Remix that signals the application's semaphores again on the graphics queue; the approach above is the same idea applied to the effect submit.
Sketch (not tested inside ReShade):A binary semaphore can be waited on and signaled again by the same batch, so this is valid Vulkan.
What I did not verify
Also useful: a per-VkQueue mutex around the application's vkQueueSubmit* and the layer's own submit (ReShade already has queue_impl->_mutex), and a one-line log of the device extensions, the swapchain create flags and the pNext types of the first presents, which is what made the difference between the two Protons visible.
Happy to send a patch if you think the change is worth having; I can only test it on Linux with a native Vulkan application.
I maintain a small ReShade-compatible Vulkan layer for Linux (it runs ReShade FX effects through a Vulkan layer instead of through Wine). In vkQueuePresentKHR it did what ReShade's Vulkan hook does: it submits the effect work waiting on the present's wait semaphores and then replaces the present's wait list with the semaphore the effect submit signals.
That worked with every application I tried until I ran a game through DXVK 3.1.1-97 (Proton Experimental). It creates its swapchain with VK_KHR_present_id2 / present_wait2 flags and attaches VkSwapchainPresentFenceInfoEXT and VkSwapchainPresentModeInfoEXT to every present (VK_EXT_swapchain_maintenance1). With a heavy effect chain the GPU hung within a minute or two (RTX 3060, driver 615.78.08: Xid 69 "Class Error", once Xid 109 "CTX SWITCH TIMEOUT"). The same game and effects with GE-Proton 11-7 (DXVK 3.1-19, which does not use those extensions) never hung, and a light preset on the same DXVK did not hang either.
What fixed it
Not replacing the wait list. The effect submit waits on the application's semaphores and signals the same semaphores again; the present then waits on exactly what the application handed over, and the rest of the present info (including the pNext chain with the present fence / present id) is untouched. Validation layers stay clean. With that change the same game, same Proton, same heavy preset ran without any Xid (it hung every time before).
Why it may matter for ReShade
source/vulkan/vulkan_hooks_swapchain.cpp, at the end of vkQueuePresentKHR:
// Override wait semaphores based on the last queue submit
present_info.waitSemaphoreCount = submit_info.waitSemaphoreCount;
present_info.pWaitSemaphores = submit_info.pWaitSemaphores;Sketch (not tested inside ReShade):
// in flush_immediate_command_list(wait_semaphore_info):
if (wait_semaphore_info->waitSemaphoreCount != 0) {
submit_info.signalSemaphoreCount = wait_semaphore_info->waitSemaphoreCount;
submit_info.pSignalSemaphores = wait_semaphore_info->pWaitSemaphores; // re-signal what we waited on
}
// ...and do not override present_info.pWaitSemaphores afterwardsWhat I did not verify
- I did not run ReShade's own Vulkan layer. ReShade under Wine/Proton hooks D3D11/DXGI and never reaches this code, so this only concerns native Vulkan applications that use the new present extensions.
- I have no native Vulkan test case that uses VK_EXT_swapchain_maintenance1 with ReShade. Whether it also hangs there may depend on the effect chain being heavy.
Also useful: a per-VkQueue mutex around the application's vkQueueSubmit* and the layer's own submit (ReShade already has queue_impl->_mutex), and a one-line log of the device extensions, the swapchain create flags and the pNext types of the first presents, which is what made the difference between the two Protons visible.
Happy to send a patch if you think the change is worth having; I can only test it on Linux with a native Vulkan application.
Please Log in or Create an account to join the conversation.
