Just realized how generic the setback post was

Add comments

Wow. I just realized how generic and unspecific the “setback” post was. Anyways, if you are curious, I kept running into a simple setback that dealt with number precision.

The symptoms were weird. I divided a number by something like 7 using code. I then used that result as an increment in a for loop. However, there would be a weird amount of missing leftovers from the increment (like the end result was off by 4-6 )!

I would run my code line-by-line, and the math would completely check out! So, I figured out the precision issue when I took the total of whatever I was dividing with calc.exe, and was getting the correct answer, but with some stuff after the decimal. So, I took the whole number and multiplied by whatever I divided by, and lo and behold! The total – (whole * whatever) = how much I was off by. After changing from int to double, the math worked out near-perfect.

So, I learned yet another lesson about precision.

Posted on April 21st 2012 in Uncategorized

Leave a Reply