Compare Errors in Microsoft Maths

I have a library function RoundDouble() which takes a double value and rounds to 2DP. This is mostly use for display and other applications where trimming is required, but I have used it to compare values with strange results.

I use Visual Studio 10 Professional to build solutions.

The test :

if( RoundDouble(a) > RoundDouble(b) )

frequently returns TRUE, even when both values are identical BOTH rounded AND before rounding. I have of course used debugging to test the values being compared.

If I use :

if( ( RoundDouble(a) - RoundDouble(b) ) > (double) 0.00001 )

This works without fail on the same data. Based on the number of DPs displayed when debugging, this test should not make a difference.

Is there something I can do that will remove this anomaly?

No, there is nothing you can do to fix that - that is an inherent property of floating point math. If you need exact precision with your calculations, don't use a floating point type.
I wish I could use that argument with our application software - sorry - it just makes mistakes!!! (lol)

Based on what you said, the RoundDouble() function uses longs

long result;

if( val >= (double) 0 )
result = (long) ( ( val * (double) 100 ) + (double) 0.5 );
result = (long) ( ( val * (double) 100 ) - (double) 0.5 );

return ( (double) result / (double) 100 );

I assume that if RoundDouble was changed to RoundDoubleToLong() to return long, this would fix the problem.

The above code would change to returning result.

Casting away the decimal places is definitely not ideal. There are existing C and C++ libraries for exact math with decimal types, for example:
Topic archived. No new replies allowed.