VK_ERROR_OUT_OF_HOST_MEMORY from vk.AllocateMemory

  • Posts: 83
1 month 2 days ago #1 by Niko of Death
Since Vulkan support was first implemented, I have been unable to use it, getting
Failed to initialize Vulkan runtime environment on runtime [pointer]
errors the entire time, not crashing the game(s), but leaving reshade completely non functional.
I finally decided to try and see what was causing this myself (assuming that this error was what is causing reshade not to work), and (through a lot of trial and error) was able to figure out the breaking point.
When vkCreateSwapchainKHR calls on_init(*pSwapchain, *pCreateInfo, hwnd), create_image() is called 3 times, once for the back buffer, once for the depth stencil, and once for the empty depth image. The first run (for the back buffer) works fine, however, on the next run, at
check_result(vk.AllocateMemory(_device, &alloc_info, nullptr, &mem)) VK_NULL_HANDLE;
AllocateMemory returns -1 (VK_ERROR_OUT_OF_HOST_MEMORY) rather than 0 (VK_SUCCESS), causing create_image (and thus on_init) to return VK_NULL_HANDLE
Various variables I collected to try and determine what the issue was (obviously variable from run to run)
back buffer AllocateMemory (VK_SUCCESS):
 _device = 000002A437EBB800
&alloc_info = 000000DE4D5AD970
alloc_info.sType = 5
alloc_info.pNext = 000000DE4D5AD990
alloc_info.allocationSize = 9437184
alloc_info.memoryTypeIndex = 0
_memory_props.memoryHeaps.size = 8321499136
_memory_props.memoryTypeCount = 4
_memory_props.memoryType.propertyFlags VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT = 1
&mem = 000000DE4D5AD9C8
depth stencil AllocateMemory (VK_ERROR_OUT_OF_HOST_MEMORY)
_device = 000002A437EBB800
&alloc_info = 000000DE4D5AD970
alloc_info.sType = 5
alloc_info.pNext = 000000DE4D5AD990
alloc_info.allocationSize = 2555912
alloc_info.memoryTypeIndex = 0
_memory_props.memoryHeaps.size = 8321499136
_memory_props.memoryTypeCount = 4
_memory_props.memoryType.propertyFlags VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT = 1
&mem = 000000DE4D5AD9C8
Looking at the documentation for AllocateMemory ( www.khronos.org/registry/vulkan/specs/1....kAllocateMemory.html ), there's no apparent reason for it to fail like this with the information I have, leaving me stuck on how to fix the issue.

Please Log in or Create an account to join the conversation.

  • Posts: 83
1 month 1 day ago #2 by Niko of Death
For some reason I can't edit, forgot to put system info:
Windows 10 1909
RX 580 (20.1.3 drivers)
i5 3570k

Please Log in or Create an account to join the conversation.

  • Posts: 3635
1 month 1 day ago #3 by crosire
Is it this specific allocation fails or does it just generally fail after a certain amount of allocations (e.g. does the next allocation still fail if you comment this one out)? Is this true everywhere, including the test app (Debug App target in Visual Studio), or just specific games?
The following user(s) said Thank You: Niko of Death

Please Log in or Create an account to join the conversation.

  • Posts: 83
1 month 1 day ago #4 by Niko of Death

crosire wrote: Is it this specific allocation fails or does it just generally fail after a certain amount of allocations (e.g. does the next allocation still fail if you comment this one out)? Is this true everywhere, including the test app (Debug App target in Visual Studio), or just specific games?

It's only this specific allocation, commenting out the allocation (or just running it on its own without the VK_SUCCESS check) gets it to work (i.e. log file shows a successful runtime return), but the game crashes, after
Loading image files for textures ...
with no shaders installed, or
Successfully loaded "DisplayDepth.fx".
when I have it included.
It's the case for the games I've tested, which are No Mans Sky, Doom 2016, RAGE 2.
Using the debug app, I get an assertion failure earlier:
File: source\dll_main.cpp
Line: 549

Expression: res == VK_SUCCESS || res == VK_SUBOPTIMAL_KHR
Looking up the error code this doesn't seem fatal, continuing from there I get an access violation exception:
Exception thrown at 0x0000000000000000 in ReShade64.exe: 0xC0000005: Access violation executing location 0x0000000000000000.
at line 556 (VkCommandPool cmd_alloc = VK_NULL_HANDLE;), not sure what I can do from there.

Please Log in or Create an account to join the conversation.

  • Posts: 83
1 month 1 day ago #5 by Niko of Death
Also, should note that with the error check for the allocation removed, that allocation is the only one that gets the error code returned (2 more run after, both with VK_SUCCESS)

Please Log in or Create an account to join the conversation.

  • Posts: 3635
1 month 10 hours ago #6 by crosire
Debug app is failing on device creation because you are missing the Vulkan SDK most likely and it's trying to pull in a couple of debugging layers that are part of that SDK.
The allocation failing sounds like a driver bug. This particular error code should not be returned under normal behavior. You can try this: github.com/crosire/reshade/commit/5e72f0...c9b05f22395540cfd2fc and hope that the driver bug is not triggered with that usage flag removed (which was superflous anyway).
The following user(s) said Thank You: Niko of Death

Please Log in or Create an account to join the conversation.

  • Posts: 83
1 month 6 hours ago #7 by Niko of Death

crosire wrote: Debug app is failing on device creation because you are missing the Vulkan SDK most likely and it's trying to pull in a couple of debugging layers that are part of that SDK.
The allocation failing sounds like a driver bug. This particular error code should not be returned under normal behavior. You can try this: github.com/crosire/reshade/commit/5e72f0...c9b05f22395540cfd2fc and hope that the driver bug is not triggered with that usage flag removed (which was superflous anyway).

No luck, unfortunately. Assuming it is a driver bug, it's strange that it has both persisted throughout countless driver updates (I've been trying to use reshade in vulkan since the support was first pushed to master) and seemingly not cropped up for others with the same graphics card. I've even done multiple clean installs. Also, I do still have the Vulkan SDK installed (since it was needed to compile reshade at one point), is there a specific version needed? I have 1.1.114.0.
Is there anywhere to go from here? If it helps, here's my vulkaninfo readout: pastebin.com/DxWpiRc3

Please Log in or Create an account to join the conversation.

  • Posts: 3635
3 weeks 3 days ago - 3 weeks 3 days ago #8 by crosire
Sooo, it turns out your GPU does not support the D24S8 format, which I find really surprising, considering that the card has to support it in DX. But not Vulkan apparently: vulkan.gpuinfo.org/displayreport.php?id=7694#formats . And that's why that image creation fails. Looks like I need to find a different format.

Please Log in or Create an account to join the conversation.

  • Posts: 3635
3 weeks 3 days ago #9 by crosire

Please Log in or Create an account to join the conversation.

  • Posts: 83
3 weeks 2 days ago #10 by Niko of Death
Compiled the latest master, can confirm this fixes the issue and reshade finally works in Vulkan for me! Thanks.

Please Log in or Create an account to join the conversation.