HBAO+ support ?

6 years 2 months ago - 6 years 2 months ago #1 by Boulotaur2024
HBAO+ support ? was created by Boulotaur2024
Hi everyone, I wanted to PM Crosire but I couldn't find the PM button :/

I've been playing around with the idea of integrating HBAO+ library into whatever injector is available. I guess it wouldn't be very useful to NVIDIA users but for us AMD users it would be convenient to have that feature at hand. Note that It would only work for DX11 though.

Reshade source-code not being available I had to dig back my old messy outdated injector and modify it. Well the library is actually really really easy to integrate. You do have to drop the HBAO+ library into the same directory as the targeted game. I actually got it to work on a color texture. It does work but as you can guess that's certainly not the proper texture to feed the shader with of course. You do need the depth texture.

And the problem is that I can't seem to be able to write depth into my own custom depth texture. I'm not even getting a full white texture that would indicate the depth values are not linearized but a black texture, which means... that I'm lost. I mean it could be a lot of things. I can't even use the D3D11 debug runtime since that project still use the old DX SDK from june 2010 which doesn't work anymore for Windows 8/10

I don't know if it's ok to pretend we're in gamedev forums but here's the depth texture/DSV/SRV creation code :

	td.Width = pDesc.BufferDesc.Width;
	td.Height = pDesc.BufferDesc.Height;
	td.MipLevels = 1;
	td.ArraySize = 1;
	td.SampleDesc.Count = 1;
	td.SampleDesc.Quality = 0;
	td.Usage = D3D11_USAGE_DEFAULT;
	td.CPUAccessFlags = 0;
	td.MiscFlags = 0;
	hr = pDevice->CreateTexture2D(&td, 0, &_depthMap);

	ZeroMemory(&dsvd, sizeof(dsvd));
	dsvd.Format = DXGI_FORMAT_D32_FLOAT;
	dsvd.ViewDimension = D3D11_DSV_DIMENSION_TEXTURE2D;
	dsvd.Texture2D.MipSlice = 0;
	hr = pDevice->CreateDepthStencilView(_depthMap, &dsvd, &_dmapDSV);

	srvd.Format = DXGI_FORMAT_R32_FLOAT;
	srvd.ViewDimension = D3D11_SRV_DIMENSION_TEXTURE2D;
	srvd.Texture2D.MipLevels = td.MipLevels;
	srvd.Texture2D.MostDetailedMip = 0;
	hr = pDevice->CreateShaderResourceView(_depthMap, &srvd, &_dmapSRV);

I'm injecting that depthstencil view into a hooked OMSetRenderTargets() :

void ID3D11devcontext::OMSetRenderTargets(UINT NumViews, ID3D11RenderTargetView *const *ppRenderTargetViews, ID3D11DepthStencilView *pDepthStencilView) {
	if (pDepthStencilView) {
		if (DepthMap::get().getdmapDSV()) {
			return pdevContext->OMSetRenderTargets(NumViews, ppRenderTargetViews, DepthMap::get().getdmapDSV()); // DepthMap::get().getdmapDSV() is an accessor to _dmapDSV created above
		else {
			return pdevContext->OMSetRenderTargets(NumViews, ppRenderTargetViews, pDepthStencilView);
	else {
		return pdevContext->OMSetRenderTargets(NumViews, ppRenderTargetViews, pDepthStencilView);

At this stage I'm hoping the "host game" has rendered scene and written depth into my own depth texture since I hacked into OMSetRenderTargets to force mine instead of the game's.
But when I try to output it to the screen it's all desperately black. Maybe it needs to be unbound first... but I think I've done everything described in that article ("Reading the Depth-Stencil Buffer as a Texture" section) right. So I'm confused.

So I was thinking if you guys and especially Crosire struggled in the past with that as well an insight would be warmly welcome.
(I forgot to mention that I'm using a DX11 NVIDIA sample that 100% works with Reshade depth access to make sure I get things right)
Last edit: 6 years 2 months ago by Boulotaur2024.
The following user(s) said Thank You: piltrafus

Please Log in or Create an account to join the conversation.

6 years 2 months ago #2 by kingeric1992
Replied by kingeric1992 on topic HBAO+ support ?
The "+" can already be done in reshade since it is capable of 4xMRT, the only thing it required in dx11 is texture array, which probably doesn't matters on performance wise compares to normal texture read. (don't know about d3d9.dll though?)

Please Log in or Create an account to join the conversation.

6 years 2 months ago #3 by crosire
Replied by crosire on topic HBAO+ support ?
Replacing the depthstencil view isn't as easy as it sounds. Games tend to use a whole range of different buffers for different usages, which means replacing them all with the same buffer doesn't work, they are too different for that (can be of different size, different format, cleared at various times between rendering etc.).
First thing is to find the one depthstencil view that contains the whole scene depth data (if it exists, sometimes even that data is splitted across multiple depthstencils, unfortunately). Next make sure to only replace exactly that depthstencil with one that is of a format which is readable from shaders. Also important is to carefully unbind it from the render pipeline before actually using it in a shader, since it's impossible to read and write to the same buffer at the same time.

Please Log in or Create an account to join the conversation.

6 years 2 months ago - 6 years 2 months ago #4 by Kleio420
Replied by Kleio420 on topic HBAO+ support ?
maybe i read the TOS wrong but if your using this in anyway that generates profit(donations) i was under the impression hbao+ was not allowed to be used. Please feel free to correct me in the most crappy way possible crosire/anyone else cause you like to do it , i just dont feel like logging in to check something like that
Last edit: 6 years 2 months ago by Kleio420.

Please Log in or Create an account to join the conversation.

6 years 2 months ago #5 by Marty McFly
Replied by Marty McFly on topic HBAO+ support ?
Well HBAO+ is closed source AFAIK and ReShade isn't supposed to hook dlls so it's probably not going to happen anyways. Plus some other AO implementations look better anyways.
The following user(s) said Thank You: Kleio420

Please Log in or Create an account to join the conversation.