Re: print() precision

From: Dave Allured - NOAA Affiliate <dave.allured_at_nyahnyahspammersnyahnyah>
Date: Wed Jul 31 2013 - 13:18:09 MDT

All,

It is important to display the maximum possible number of significant
digits when dealing with precision issues. NCL's default print
functions do not generally show all digits necessary for full
precision. The following sprintf format codes should display the
necessary significant digits in all possible cases, except NaN's of
course:

xs = "0.407115567890123456789"
xf = stringtofloat (xx)
xd = stringtodouble (xx)
print ("xs = " + xx)
print ("xf = " + sprintf ("%.9g", xf))
print ("xd = " + sprintf ("%.17g", xd))

Output:
(0) xx = 0.407115567890123456789
(0) fxx = 0.407115579
(0) dxx = 0.40711556789012343

Please note that the number of digits necessary to guarantee UNIQUE
displayed numbers is MORE than the accuracy of the actual numbers. In
the single precision case above, 9 non-zero digits are displayed, but
only 7 are accurate. In the double precision case, 17 digits are
displayed, but only 16 are accurate.

The "g" format is necessary for very small or large numbers that need
to be displayed with an exponent. The values of 9 digits for single
precision, and 17 digits for double precision were determined by
previous authors to be the minimum necessary to distinguish two
adjacent numbers in all cases. These minimums are documented in IEEE
754 floating point standards, Wikipedia (thanks Arne), and other
references.

http://en.wikipedia.org/wiki/Single-precision_floating-point_format
http://en.wikipedia.org/wiki/Double-precision_floating-point_format

If anyone finds cases in NCL where sprintf "%.9g" and "%.17g" do not
correctly show unique strings for different floating point numbers, or
you have a better way to show full precision, please let us know.

--Dave

On Wed, Jul 31, 2013 at 10:07 AM, Dennis Shea <shea@ucar.edu> wrote:
> xf = 0.4071155678
> xd = 0.4071155678d
>
> print("xf="+xf +" : "+sprintf("%13.6f", xf))
> print("xd="+xd +" : "+sprintf("%13.10f", xd))
>
> ==============
>
> (0) xf=0.407116 : 0.407116
> (0) xd=0.407116 : 0.4071155678
>
>
> On 7/31/13 9:57 AM, Wei Huang wrote:
>> Fan,
>>
>> Try
>>
>> print(0.407115567890123456789d)
>>
>> Wei
>>
>> huangwei@ucar.edu
>> VETS/CISL
>> National Center for Atmospheric Research
>> P.O. Box 3000 (1850 Table Mesa Dr.)
>> Boulder, CO 80307-3000 USA
>> (303) 497-8924
>>
>>
>> On Jul 31, 2013, at 9:41 AM, Fan Fang <Fan.Fang@nasa.gov> wrote:
>>
>>> It appear print() only does floats. IS there an easy way to print out
>>> just what is fed?
>>>
>>> -Fan
>>>
>>> begin
>>> print(0.4071155678)
>>> end
>>>
>>> NCAR Command Language Version 5.1.1
>>> (0) 0.4071156
_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Wed Jul 31 13:18:20 2013

This archive was generated by hypermail 2.1.8 : Thu Aug 01 2013 - 15:55:03 MDT