Issue with create debug device using latest src

  • pyromuffin
  • Topic Author
More
5 years 9 months ago - 5 years 9 months ago #1 by pyromuffin Issue with create debug device using latest src was created by pyromuffin
Hi, I'm hacking away on a fork of Reshade, and I'm currently unable to create a debug device. I'm wondering if this functionality is busted, because debug builds are crashing immediately, and it's due to the call to

HRESULT hr = reshade::hooks::call(&D3D11CreateDeviceAndSwapChain)(pAdapter, DriverType, Software, Flags, pFeatureLevels, FeatureLevels, SDKVersion, nullptr, nullptr, ppDevice, &FeatureLevel, nullptr);

in d3d11.cpp, and the call stack looks like this
 	0000000000000000()	Unknown
 	d3d11_3SDKLayers.dll!00007ffb61f1c248()	Unknown
 	d3d11.dll!00007ffb6f277e7e()	Unknown
 	d3d11.dll!00007ffb6f277c9f()	Unknown
 	d3d11.dll!00007ffb6f2ad292()	Unknown
 	d3d11.dll!00007ffb6f2acf85()	Unknown
 	d3d11.dll!00007ffb6f2acd84()	Unknown
 	d3d11.dll!00007ffb6f26dfe4()	Unknown
 	d3d11.dll!00007ffb6f26720b()	Unknown
 	d3d11.dll!00007ffb6f265db9()	Unknown
 	d3d11.dll!00007ffb6f265a38()	Unknown
>	dxgi.dll!D3D11CreateDeviceAndSwapChain(IDXGIAdapter * pAdapter, D3D_DRIVER_TYPE DriverType, HINSTANCE__ * Software, unsigned int Flags, const D3D_FEATURE_LEVEL * pFeatureLevels, unsigned int FeatureLevels, unsigned int SDKVersion, const DXGI_SWAP_CHAIN_DESC * pSwapChainDesc, IDXGISwapChain * * ppSwapChain, ID3D11Device * * ppDevice, D3D_FEATURE_LEVEL * pFeatureLevel, ID3D11DeviceContext * * ppImmediateContext) Line 32	C++
 	dxgi.dll!D3D11CreateDevice(IDXGIAdapter * pAdapter, D3D_DRIVER_TYPE DriverType, HINSTANCE__ * Software, unsigned int Flags, const D3D_FEATURE_LEVEL * pFeatureLevels, unsigned int FeatureLevels, unsigned int SDKVersion, ID3D11Device * * ppDevice, D3D_FEATURE_LEVEL * pFeatureLevel, ID3D11DeviceContext * * ppImmediateContext) Line 19	C++
 	NvCamera64.dll!00007ffb4323fbea()	Unknown
 	NvCamera64.dll!00007ffb43126e99()	Unknown
 	NvCamera64.dll!00007ffb43127762()	Unknown
 	NvCamera64.dll!00007ffb430ff51f()	Unknown
 	[External Code]	
 	dxgi.dll!D3D11CreateDeviceAndSwapChain(IDXGIAdapter * pAdapter, D3D_DRIVER_TYPE DriverType, HINSTANCE__ * Software, unsigned int Flags, const D3D_FEATURE_LEVEL * pFeatureLevels, unsigned int FeatureLevels, unsigned int SDKVersion, const DXGI_SWAP_CHAIN_DESC * pSwapChainDesc, IDXGISwapChain * * ppSwapChain, ID3D11Device * * ppDevice, D3D_FEATURE_LEVEL * pFeatureLevel, ID3D11DeviceContext * * ppImmediateContext) Line 32	C++
 	dxgi.dll!D3D11CreateDevice(IDXGIAdapter * pAdapter, D3D_DRIVER_TYPE DriverType, HINSTANCE__ * Software, unsigned int Flags, const D3D_FEATURE_LEVEL * pFeatureLevels, unsigned int FeatureLevels, unsigned int SDKVersion, ID3D11Device * * ppDevice, D3D_FEATURE_LEVEL * pFeatureLevel, ID3D11DeviceContext * * ppImmediateContext) Line 19	C++

with an access violating trying to read from address 0.

it's curious that we're hitting D3D11CreateDeviceAndSwapChain twice in that callstack, a second time after passing through NvCamera64.dll ... which I think is nvidia ansel? I wonder if this has something to do with that. Maybe I'll try it on a non-ansel game.

EDIT:
Disabled ansel, and NvCamera64.dll dropped out of the callstack, now we only have one call to D3D11CreateDeviceAndSwapChain() but it still crashes at the same place in d3d11_3SDKLayers.dll. Hmmm.

EDIT 2:
The debug app seems to work with the debug layer. I'll compare and contrast the args.

EDIT 3:
Copying the args that the debug app uses doesn't help. Same crash result. Weird.

EDIT 4:
grabbing the symbols from the MS symbol server gives function names to the callstack. More to investigate:
>	d3d11_3SDKLayers.dll!NDebug::CDevice::LLOCompleteLayerConstruction(void)	Unknown
 	d3d11.dll!NOutermost::CDevice::FinalConstruct(struct NOutermost::CDevice::TConstructorArgs const &)	Unknown
 	d3d11.dll!TComObject<class NOutermost::CDevice>::TComObject<class NOutermost::CDevice>(void *,struct NOutermost::CDevice::TConstructorArgs const &,struct _GUID const &,void * *)	Unknown
 	d3d11.dll!TComObject<NOutermost::CDevice>::CreateInstance()	Unknown
 	d3d11.dll!D3D11CreateLayeredDevice()	Unknown
 	d3d11.dll!D3D11CoreCreateLayeredDevice()	Unknown
 	d3d11.dll!D3D11RegisterLayersAndCreateDevice()	Unknown
 	d3d11.dll!D3D11CoreCreateDevice()	Unknown
 	d3d11.dll!D3D11CreateDeviceAndSwapChainImpl()	Unknown
 	d3d11.dll!D3D11CreateDeviceAndSwapChain()	Unknown
 	dxgi.dll!D3D11CreateDeviceAndSwapChain(IDXGIAdapter * pAdapter, D3D_DRIVER_TYPE DriverType, HINSTANCE__ * Software, unsigned int Flags, const D3D_FEATURE_LEVEL * pFeatureLevels, unsigned int FeatureLevels, unsigned int SDKVersion, const DXGI_SWAP_CHAIN_DESC * pSwapChainDesc, IDXGISwapChain * * ppSwapChain, ID3D11Device * * ppDevice, D3D_FEATURE_LEVEL * pFeatureLevel, ID3D11DeviceContext * * ppImmediateContext) Line 39	C++
 	dxgi.dll!D3D11CreateDevice(IDXGIAdapter * pAdapter, D3D_DRIVER_TYPE DriverType, HINSTANCE__ * Software, unsigned int Flags, const D3D_FEATURE_LEVEL * pFeatureLevels, unsigned int FeatureLevels, unsigned int SDKVersion, ID3D11Device * * ppDevice, D3D_FEATURE_LEVEL * pFeatureLevel, ID3D11DeviceContext * * ppImmediateContext) Line 19	C++
Last edit: 5 years 9 months ago by pyromuffin.

Please Log in or Create an account to join the conversation.

  • crosire
More
5 years 9 months ago #2 by crosire Replied by crosire on topic Issue with create debug device using latest src
Just remove the "D3D11_CREATE_DEVICE_DEBUG" flag from the call in d3d11.cpp. I don't know why the debug layers cause problems for some, as I never had this problem myself, but you are not the first with this particular issue.

Please Log in or Create an account to join the conversation.

  • pyromuffin
  • Topic Author
More
5 years 9 months ago #3 by pyromuffin Replied by pyromuffin on topic Issue with create debug device using latest src
i mean yeah, that does work, but i need to create a debug device, for you know, debugging :D

i’ll continue to investigate.

Please Log in or Create an account to join the conversation.

  • pyromuffin
  • Topic Author
More
5 years 9 months ago - 5 years 9 months ago #4 by pyromuffin Replied by pyromuffin on topic Issue with create debug device using latest src
Digging through the disassembly at the crash site, and with a little help of a friend at MS ;) our hooked DXGI.dll fails to export a function called CompatValue, GetProcAddress returns null, and it crashes. I'm not an expert on DLL hooking, but it appears that we aren't exporting CompatValue. According to MSDN on GetModuleAddress: msdn.microsoft.com/en-us/library/windows...683199(v=vs.85).aspx

"if lpModuleName does not include a path and there is more than one loaded module with the same base name and extension, you cannot predict which module handle will be returned."

which might explain why sometimes it crashes, and sometimes it works. @crosire If you have an idea of how to just get our DLL to jump to that function in the regular DXGI.dll, I think that will do it. I don't know how to export a function when I don't know what the signature is, but I have barely looked into how the actual DLL hooking works. I mean, that's why I am using Reshade, and not writing my own app. :D

Thanks!

EDIT:
Changing the exports.def to export CompatValue and CompatString, and creating two functions:
HOOK_EXPORT void* WINAPI CompatValue()
{
	return reshade::hooks::call(&CompatValue);
}


HOOK_EXPORT void* WINAPI CompatString()
{
	return reshade::hooks::call(&CompatString);
}

makes things work. I can successfully create a debug device now. I can create a PR if you'd like.
Last edit: 5 years 9 months ago by pyromuffin.
The following user(s) said Thank You: bo3bber

Please Log in or Create an account to join the conversation.

  • crosire
More
5 years 9 months ago - 5 years 9 months ago #5 by crosire Replied by crosire on topic Issue with create debug device using latest src
Interesting. I guess those exports are new to Windows 10? Never knowingly saw them in a dump of my Windows 7 DXGI files. Since ReShade already exports some of the other stubs found there. Or maybe I assumed they weren't used. But yeah, please create a pull request. Nice find!
Last edit: 5 years 9 months ago by crosire.

Please Log in or Create an account to join the conversation.

  • bo3bber
More
3 years 1 month ago #6 by bo3bber Replied by bo3bber on topic Issue with create debug device using latest src
Just noting that this is still a problem today.  On my current fork of 4.9.1, I get this crash at D3D11CreateDeviceAndSwapChain when I enable the debug flag.

 >   Flags |= D3D11_CREATE_DEVICE_DEBUG;

In case it's helpful, here are exports lists for dxgi.dll
Win10l:  windows10dll.nirsoft.net/dxgi_dll.html
Win7:  www.win7dll.info/dxgi_dll.html

I also confirm that the solution from pyromuffin here still works to resolve the problem.  It doesn't look like a PR was ever made.

I'm not sure where the code should go, so I dropped it into dxgi.cpp.  

Please Log in or Create an account to join the conversation.

  • crosire
More
3 years 1 month ago #7 by crosire Replied by crosire on topic Issue with create debug device using latest src
Done now, with the proper function signature based on some quick reverse engineering + debug symbols: github.com/crosire/reshade/commit/66098e...d0f431a3f02396000e34

Please Log in or Create an account to join the conversation.

  • bo3bber
More
3 years 1 month ago #8 by bo3bber Replied by bo3bber on topic Issue with create debug device using latest src
Brilliant! Thank you so much. I'll merge in your latest changes.

Please Log in or Create an account to join the conversation.

We use cookies
We use cookies on our website. Some of them are essential for the operation of the forum. You can decide for yourself whether you want to allow cookies or not. Please note that if you reject them, you may not be able to use all the functionalities of the site.
Ok Decline
More information