Integer Error

[MattC]

Hi,

Can someone please explain why I get this error in my program while using integers (%).

The following contains the relevant part of my code.

Code: Select all

     DIM R%(0), V%(3)
      R%(0) = 20000
      V%(3) = 1200
      V%(1) = V%(3) / (1 + (R%(0) / 100000))
      PRINT V%(3) / (1 + (R%(0) / 100000))
      PRINT V%(1)

Which results in:

Code: Select all

    1000
     999

Why is it that the second part of the equation 'V%(1) = V%(3) / (1 + (R%(0) / 100000))' equals 1000, but V%(1) does not? Even changing the lines so...

Code: Select all

      DIM R%(0), V%(3)
      R%(0) = 20000
      V%(3) = 1200
      V = V%(3) / (1 + (R%(0) / 100000))
      V%(1) = V
      PRINT V%(3) / (1 + (R%(0) / 100000))
      PRINT V
      PRINT V%(1)

...does not correct it. This results in:

Code: Select all

    1000
    1000
     999

The floating point variable V seems to contain the correct value. But when moved to V%(1), the value picks up the error. I'm guessing it's to do with the inaccuracy of the floating point, but I can't work out why.

Matt

[Hated Moron]

Can someone please explain why I get this error in my program while using integers (%).

Well, for a start it's not an "error" - that's an emotive word because it implies something is wrong, which it isn't.

The reason you get that 'result' is because of rounding. Let's take it step by step. The calculation you are doing boils down to this:

Code: Select all

V = 1200 / 1.2

This sets V to a value slightly less than 1000, and when I say slightly we can see just how small the difference is as follows:

Code: Select all

V = 1200 / 1.2
PRINT 1000 - V

This prints 5.55111512E-17 so the value is very nearly 1000, but not quite.

When you PRINT a number, it is automatically rounded to the nearest value that can be represented with the precision specified (by default 9 significant figures) so of course PRINT V prints 1000.

But when you cast a value to an integer it is rounded down to the next lowest integer (technically it is truncated towards zero). So because V is very slightly less than 1000, despite the difference being infinitesimal, it is rounded down to 999.

All completely normal. For reference you might like to know just how precisely the value 1.2 can be represented as an 80-bit floating-point number. This is that closest value:

Code: Select all

1.20000000000000000004336808689942017736029811203479766845703125

[MattC]

Well, for a start it's not an "error" - that's an emotive word because it implies something is wrong, which it isn't.

My Bad!

Thanks for your explanation. Since writing this post I added a + 0.01 to the end of the calculation. This seems to force the result up by just enough to give a correct value without causing a problem anywhere else.

Thanks again.

Matt

[Hated Moron]

Since writing this post I added a + 0.01 to the end of the calculation.

I need hardly say that such a bodge is likely to bite you later. For example, do you know that it works on an ARM CPU or in a browser (both of which use 64-bit floats rather than 80-bit floats)? It would be better to fix the problem 'at source', for example if the program requires absolute precision could it be rewritten to use only integers?

[MattC]

Yes, I think you're right. A rearrangement of the formula might be better.

Matt