Re: A truncation issue with tointeger()?

Emilie,

It is not NCL, it is floating point computer math in general. In
general, floating point numbers in computing must not be considered
exact. You are starting with two numbers that do NOT have exact
representations in IEEE floating point:

ncl 0> a = -1.05
ncl 1> dvar = 0.15
ncl 2> print (sprintf("%15.12f", a))
(0) -1.049999952316
ncl 3> print (sprintf("%15.12f", dvar))
(0) 0.150000005960

The tiny discrepancies propagate into huge or unexpected errors when
you use the result in commands that may round or truncate, such as
print() and tointeger().

There are valid methods for handling round-off errors. One
possibility in your example would be to use the round() function
when trying to compute the exact number of units in a larger interval.

Further reading:
http://en.wikipedia.org/wiki/Roundoff_error
... and the references.

--Dave

On 1/6/2011 4:28 PM, Emilie Vanvyve wrote:
> Oh... Is there some kind of formatting/rounding going on with print() then? I thought that using that function would give me the exact value of the variable (111.99999 in this case then), especially since it sometimes likes displaying numbers with a fractional part I don't expect (see my other e-mail about ispan).
>
> If I can't trust that ( tmpV(1) - tmpV(0) ) / dvar = 112 exactly (some other time, it does though), I never can expect a float to convert to an integer without any similar problem. Pushing it to the limit, I could have tointeger((3.0-1.0)/1.) not equal to 2 but 1... Should I then always use the round() function except if I want a truncation to occur?
>
> Emilie
>
> On Jan 6, 2011, at 16:18, Dennis Shea wrote:
>
>> Rounding issue.
>>
>> tmpV = (/ -1.05, 15.75 /)
>> dvar = 0.15
>> tmp1 = ( tmpV(1) - tmpV(0) ) / dvar
>> print(tmp1) ; 112
>> print(sprintf("%5.2f", tmp1)) ; 112.00
>> print(sprintf("%12.5f", tmp1)) ; 111.99999
>>
>> i1 = toint(tmp1) ; 111
>> i2 = floattoint(tmp1) ; 111
>> i3 = round(tmp,3) ; 112
_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Thu Jan 6 16:50:50 2011