Welcome, Guest.
Username: Password: Remember me

TOPIC: behavior of division?

behavior of division? 10 months 1 day ago #1

What is the behavior of division "/" ?

0.03125 != 1/32 == 0 on my test.

for some reason,
float4 threshold = float4(index.x, index.x + 1/64, index.x + 1/32, index.x + 3/64);
and
float4 threshold = float4(index.x, index.x + 0.015625, index.x + 0.03125, index.x + 0.046875);
are having different result.
Last Edit: 10 months 1 day ago by kingeric1992.
The administrator has disabled public write access.

behavior of division? 10 months 1 day ago #2

  • crosire
  • crosire's Avatar
  • Offline
  • Posts: 2457
  • Thank you received: 1393
That's because "1" and "32" are integers. A divison with two integers results in an integer again (in your case zero). To fix this, at least one of the factors has to be a float, i.e. "1.0/32".
Cheers, crosire =)
The administrator has disabled public write access.

behavior of division? 10 months 1 day ago #3

Thanks.
The administrator has disabled public write access.