Re: basic math

To further illustrate Johnathan's explanation, here are two examples
of common practical approaches to dealing with minor roundoff errors.

1. Display only the number of significant digits that are
appropriate for your application. The formatting routine will round
off the displayed value appropriately. The roundoff error is still
present in the internal number, but it is not inappropriately displayed.

    print (sprintf ("%8.4f", 4-4.8))
    (0) -0.8000

2. If you need more precision, use double precision math.

    print (sprintf ("%18.14f", 4d - 4.8d))
    (0) -0.80000000000000

Caution: When doing a series of computations, roundoff errors can
become magnified. Two common situations that amplify roundoff
errors are adding a small number to a large one, as in summing a
long list of numbers; and subtracting two numbers that are very
close. Frequently these operations are hidden inside more complex
calculations.

I recommend the following further study for everyone who uses
floating point computer math for extended computing:

http://en.wikipedia.org/wiki/Floating_point
http://docs.sun.com/source/806-3568/ncg_goldberg.html

Dave Allured
CU/CIRES Climate Diagnostics Center (CDC)
http://cires.colorado.edu/science/centers/cdc/
NOAA/ESRL/PSD, Climate Analysis Branch (CAB)
http://www.cdc.noaa.gov/

Jonathan Vigh wrote:
> Hi Ruben,
> I think you're just experiencing the pitfalls of doing math on
> computers which represent floats using binary. This isn't a problem
> specific to NCL - you can find it in just about any programming
> language. If you're using single float precision, there will be some
> truncation error by about the 8th significant digit (or less depending
> on size of the number, etc.). Whether NCL's print statement shows the
> loss or not is another matter, but the error is still there.
>
> ncl 1> print(4.0)
> (0) 4
> ncl 2> print(4.000000000000013)
> (0) 4 <- the last significant digits are
> lost - these won't be accounted for in subsequent operations
> ncl 3> print(4.0000001)
> (0) 4
> ncl 4> print(4.0000001)
> (0) 4
> ncl 5> print(4.000001)
> (0) 4.000001 <- now the last significant digit is
> correctly preserved
> ncl 6> print(4.00001)
> (0) 4.00001
>
> When you do other operations like multiplication, the truncation errors
> propagate and introduce what may seem to be spurious signficant digits.
> Whether these show up depend on how NCL handles the printing of
> significant digits. In a plot, or when printing out variables, can be
> controlled in NCL by using some esoteric controls (the 'format
> conversion specification strings') which you can read more about here:
> http://www.ncl.ucar.edu/Document/Graphics/format_spec.shtml
>
> The solution is to use double precision if you need more accuracy:
>
> ncl 7> print(4d) <- how to represent a 4 with double precision
> (0) 4
>
> ncl 8> print(4.000000000000013d)
> (0) 4.000000000000013 <- represented okay with double
> precision
> ncl 9> print(4.0000000000000013d)
> (0) 4.000000000000001 <- starting to lose precision
> ncl 10> print(4.00000000000000013d)
> (0) 4
> So now try your subtraction with double precision:
>
> ncl 11> print(4d - 4.8d)
> (0) -0.7999999999999998
>
> This is a lot closer to the true value than the single precision result.
> If you need 'nice' looking values for plots then look into setting the
> format conversion specification strings (these can typically be set
> using a resource), use the round function, or the sprinti and sprintf
> routines.
>
> Hope this helps!
>
> Jonathan Vigh
> PhD Candidate
> Colorado State University
>
>
>
>
>
>
> Ruben van Hooidonk wrote:
>
>> Hi,
>>
>> When I do some basic math in NCL I get unexpected results. I was
>> wondering if I have made some mistake somewhere or if there is a
>> solution to prevent this behavior.
>>
>> The problem is illustrated by this snippet of code:
>> print(4-4.8)
>> (0) -0.8000002
>>
>> This phenomenon is not limited to subtraction:
>> print(1.2*5.3)
>> (0) 6.360001
>>
>> But it is sensitive to the value of the floats:
>> print(1.2*5.1)
>> (0) 6.12
>>
>> This happens while using ncl version 4.2.0.a035 on a PPC mac, version
>> 4.3.1
>> on an intel linux machine, and while using version 5.0.1 on an intel mac.
>>
>> Thank you,
>>
>> Ruben
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk_at_ucar.edu
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk_at_ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
ncl-talk_at_ucar.edu
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Thu Apr 10 2008 - 16:56:42 MDT