Re: strange arithmetic error

Hi Walter,
This is normal behavior for 32 bit floating point values. Float values have between 6 and 7 decimal digits worth of precision. You are trying to distinguish between two values that only differ at the 8th digit. Use doubles (64 bit floating values) if you need this level of precision.
Try

ncl 0> print ( 20110001.-20110000. )
(0) 0
ncl 1> print ( 20110001.d -20110000.d )
(0) 1
ncl 2> print ( 20110001.d -20110000. )
(0) 1

Notice that making one of the values explicitly double is sufficient. The float value gets "coerced" to double automatically during a binary operation (minus) between a double and a float.
 -dave

On Nov 20, 2012, at 11:31 AM, Walter Hannah wrote:

> I've come across some strange arithmetic in NCL (6.1 beta). Here's a copy and paste from a terminal window to demonstrate the problem:
>
> Copyright (C) 1995-2012 - All Rights Reserved
> University Corporation for Atmospheric Research
> NCAR Command Language Version 6.1.0-beta
> The use of this software is governed by a License Agreement.
> See http://www.ncl.ucar.edu/ for more details.
> ncl 0> print( 20110001-20110000 )
> (0) 1
> ncl 1> print( 20110001.-20110000. )
> (0) 0
> The second answer should obviously be 1.0, but its not!!!! So when using floating point numbers there seems to besome weird round off problem or something. Am I missing something here? I tested some other things and got other strange results like this one:
>
> ncl 5> print( 2011.0001-2011. )
> (0) 0.0001220703
>
>
> Should I roll back a version and see if its still happens?
>
> Walter Hannah
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Tue Nov 20 11:57:34 2012