LUT is limited to 256 x 64

Anything higher than width of 16384 will result in compile fail. Tested in Guild Wars 2 on a laptop with a 2020 Intel IGPU model 

Well yes, that's the texture dimension limit ( docs.microsoft.com/en-us/windows/win32/d...-11-resources-limits , this is true in the other APIs as well). But there is a very big gap between 256 and 16384 that can be used.

Damn my lut is much much bigger cuz it contains all possible rgb colors. is there any way to use it in reshade

I don't know what your goal is but there is no such thing as all possible rgb colors. The values you are working with are limited by the colorspace's primaries and the number of steps defined by the bit depth. The output is limited to the display colorspace the content is purposed for.

In regards to LUT resolution there is plenty of data to go with for any task because a LUT will interpolate any value inbetween. A higher resolution will only improve accuracy, not magically give more colors.

What you may be looking for is an interpolation method. I don't know the reshade details but it might be trilinear interpolation by default for performance reasons but if tetrahedral is possible that would be an improvement accuracy wise.