5.0.2 mul(matrix, vector) function throws an error

In 5.0.2 ReShade (dxgi), function mul(matrix3x2, vector3) throws an error, whereas it worked as expected in 4.9.1, resulting in vector2.

What you are trying to do is mathematically impossible and ReShade 5.0 now detects this (this didn't work in 4.9.1 either, the compiler just didn't always issue an error and instead did funky stuff like trying to reinterpret the 3x2 matrix as 2x3 which doesn't do what you'd expect or want).
A matrix multiplication of 3x2 * 3x1 is not defined, but 1x3 * 3x2 is, so you can solve this by switching the arguments to mul(vector3, matrix3x2).

Matrix multiplication would work if vector3 was interpreted as 1x3 matrix, not 3x1. Such that mul(matrix3x2, vector3) = vector2 results in 3x2 * 1x3 multiplication.
Code for that would look something like this:

No?. For matrix multiplication to be valid the number of columns in the first matrix must be equal to the number of rows in the second matrix (see also en.wikipedia.org/wiki/Matrix_multiplication ). With 3x2 * 1x3, that is not the case either (2 != 1).

You are probably thinking of 2x3 * 3x1. That's valid, can call that via mul(matrix2x3, vector3). Or as said do mul(vector3, matrix3x2).

Thanks for the response. Column*row or row*column is an arbitrary choice, depending on convention. It's called row-major or column-major order.

The following:gives same result as proposed:,
where
This is how I fixed the error, using matrix3x3 and swizzle.
Since it works, I propose convention, where matrix vector components (accessed by index) represent rows. And vector on the right side of mul function is transposed to colum-vector. Therefore mul operation performs row*column multiplication, resulting in an column vector. At first glace it should go along DirectX convention, but I have not confirmed that.[/code][/code][/code]

I'm aware, but it's not a choice you have in ReShade: In ReShade FX all matrices are row-major by definition. That's it. Their storage format cannot be changed to be column-major, like in HLSL with "row_major" and "column_major" keywords (notice those keywords are intentionally missing in ReShade FX). This is done to simplify translation between the various APIs and uniform storage updates. Similarly all matrices are logically expected to be row-major in operations and as such only those intrinsics that make sense for row-major matrices/vectors are available.