Re: basic math

Hi Ruben,
    I think you're just experiencing the pitfalls of doing math on
computers which represent floats using binary. This isn't a problem
specific to NCL - you can find it in just about any programming
language. If you're using single float precision, there will be some
truncation error by about the 8th significant digit (or less depending
on size of the number, etc.). Whether NCL's print statement shows the
loss or not is another matter, but the error is still there.

ncl 1> print(4.0)
(0) 4
ncl 2> print(4.000000000000013)
(0) 4 <- the last significant digits are
lost - these won't be accounted for in subsequent operations
ncl 3> print(4.0000001)
(0) 4
ncl 4> print(4.0000001)
(0) 4
ncl 5> print(4.000001)
(0) 4.000001 <- now the last significant digit is
correctly preserved
ncl 6> print(4.00001)
(0) 4.00001

When you do other operations like multiplication, the truncation errors
propagate and introduce what may seem to be spurious signficant digits.
Whether these show up depend on how NCL handles the printing of
significant digits. In a plot, or when printing out variables, can be
controlled in NCL by using some esoteric controls (the 'format
conversion specification strings') which you can read more about here:
http://www.ncl.ucar.edu/Document/Graphics/format_spec.shtml

The solution is to use double precision if you need more accuracy:

ncl 7> print(4d) <- how to represent a 4 with double precision
(0) 4

ncl 8> print(4.000000000000013d)
(0) 4.000000000000013 <- represented okay with double
precision
ncl 9> print(4.0000000000000013d)
(0) 4.000000000000001 <- starting to lose precision
ncl 10> print(4.00000000000000013d)
(0) 4

So now try your subtraction with double precision:

ncl 11> print(4d - 4.8d)
(0) -0.7999999999999998

This is a lot closer to the true value than the single precision result.
If you need 'nice' looking values for plots then look into setting the
format conversion specification strings (these can typically be set
using a resource), use the round function, or the sprinti and sprintf
routines.

Hope this helps!

Jonathan Vigh
PhD Candidate
Colorado State University

Ruben van Hooidonk wrote:

>Hi,
>
>When I do some basic math in NCL I get unexpected results. I was
>wondering if I have made some mistake somewhere or if there is a
>solution to prevent this behavior.
>
>The problem is illustrated by this snippet of code:
>print(4-4.8)
>(0) -0.8000002
>
>This phenomenon is not limited to subtraction:
>print(1.2*5.3)
>(0) 6.360001
>
>But it is sensitive to the value of the floats:
>print(1.2*5.1)
>(0) 6.12
>
>This happens while using ncl version 4.2.0.a035 on a PPC mac, version 4.3.1
>on an intel linux machine, and while using version 5.0.1 on an intel mac.
>
>Thank you,
>
>Ruben
>_______________________________________________
>ncl-talk mailing list
>ncl-talk_at_ucar.edu
>http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>

_______________________________________________
ncl-talk mailing list
ncl-talk_at_ucar.edu
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Thu Apr 10 2008 - 11:55:53 MDT