# Re: lost precision with string2float?

From: Dennis Shea <shea_at_nyahnyahspammersnyahnyah>
Date: Wed Jul 31 2013 - 10:01:43 MDT

xx = "2.712245356"
print("xx= "+xx)
print("------")

fxx = stringtofloat(xx)
dxx = stringtodouble(xx)
print("print default: fxx= "+fxx)
print("print sprintf: fxx= "+sprintf("%13.6f", fxx))
print("print default: dxx= "+dxx)
print("print sprintf: dxx= "+sprintf("%13.10f", dxx))
print("------")

FXX = tofloat(xx)
DXX = todouble(xx)
print("print default: FXX= "+FXX)
print("print sprintf: FXX= "+sprintf("%13.6f", FXX))
print("print default: DXX= "+DXX)
print("print sprintf: DXX= "+sprintf("%13.10f", DXX))

==================================
(0) xx= 2.712245356
(0) ------
(0) print default: fxx= 2.71225
(0) print sprintf: fxx= 2.712245
(0) print default: dxx= 2.71225
(0) print sprintf: dxx= 2.7122453560
(0) ------
(0) print default: FXX= 2.71225
(0) print sprintf: FXX= 2.712245
(0) print default: DXX= 2.71225
(0) print sprintf: DXX= 2.7122453560

On 7/31/13 9:52 AM, é›·éœ† wrote:
> Wei,
> That is great. My results are exactly as yours.
> Helps from you and Arne are so appreciated. Without your help, I won't consider the differences in float and double, at least for quite some time, since,in my original print, they gave the same results.
> Thanks again.
> Ting
> Subject: Re: [ncl-talk] lost precision with string2float?
> From: huangwei@ucar.edu
> Date: Wed, 31 Jul 2013 09:01:45 -0600
> CC: arne.melsom@met.no; ncl-talk@ucar.edu
> To: leiting2002@hotmail.com
>
> Ting,
> It is probably the "print" function have some issue,
> try:
> xx="2.712245356"fxx=stringtofloat(xx)dxx=stringtodouble(xx)print(fxx)print(dxx)
> It will show you more accuracy.
> you may use the toxxx functions as below.
> fxx:=tofloat(xx)dxx:=todouble(xx)print(fxx)print(dxx)
> Wei
>
> huangwei@ucar.eduVETS/CISL
> National Center for Atmospheric Research
> P.O. Box 3000 (1850 Table Mesa Dr.)
> Boulder, CO 80307-3000 USA
> (303) 497-8924
>
>
>
> On Jul 31, 2013, at 6:35 AM, \$)A@Wv* <leiting2002@hotmail.com> wrote:Arne,
> Sorry for my stupid mistake:). With the corrected one:
> ^^
> [tlei:/lustre/scratch/tlei/hybrid-GSI/tlei-postprocessing/dr-obfits]\$ cat deb.ncl
>
> begin
> xx="2.712245356"
> fxx=stringtofloat(xx)
> dxx=stringtodouble(xx)
> print("fxx= "+fxx)
> print("dxx= "+dxx)
> end
> Vv
> It gave the same result.
> Thank you .
> Ting
>
>> Date: Wed, 31 Jul 2013 12:22:23 +0000
>> From: arne.melsom@met.no
>> To: leiting2002@hotmail.com
>> CC: ncl-talk@ucar.edu
>> Subject: Re: [ncl-talk] lost precision with string2float?
>>
>> Hi,
>> I don't know why you get the same result with stringtofloat and stringtodouble. But I note that your 'Kraken' example has an error
>> xx="2.712245356"
>> fxx=stringtofloat(xx)
>> dxx=stringtofloat(xx)
>> Here, you use stringtofloat for both fxx and dxx. If there really is a problem with stringtodouble, I hope that someone else can help you; I don't know what may be wrong. But I can assure that, with 12 digits, stringtofloat will always give you a round-off error.
>> Best regards,
>> Arne
>>
>> ----- Original Message -----
>>> Hi, Arne,
>>> Thank you so much for your suggestions.
>>> I had suspected on the impact of the difference between stringtofloat
>>> and stringtodouble and found no differences. Just now, I just tried it
>>> again. seems the same. In the previous example, I changed it into :
>>> ^^
>>> fxx = stringtofloat(str_get_field(str_data(2:11),10 ," "))
>>> dxx = stringtodouble(str_get_field(str_data(2:11),10 ," "))
>>> print("output fxx "+ fxx)
>>> print("output dxx "+ dxx)
>>> VV
>>> It gave the same results. In case it is machine dependent, I tried it
>>> on another machine (Kraken).
>>> The test ncl file is
>>> [tlei:/lustre/scratch/tlei/hybrid-GSI/tlei-postprocessing/dr-obfits]\$
>>> cat deb.ncl
>>> ^^
>>> "/sw/xt-cle3.1/ncl/5.0.0/sles11.1_pgi11.6.0/lib/ncarg/nclscripts/csm/gsn_code.ncl"
>>> "/sw/xt-cle3.1/ncl/5.0.0/sles11.1_pgi11.6.0/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
>>> "/sw/xt-cle3.1/ncl/5.0.0/sles11.1_pgi11.6.0/lib/ncarg/nclscripts/csm/contributed.ncl"
>>> "/sw/xt-cle3.1/ncl/5.0.0/sles11.1_pgi11.6.0/lib/ncarg/nclscripts/csm/shea_util.ncl"
>>> begin
>>> xx="2.712245356"
>>> fxx=stringtofloat(xx)
>>> dxx=stringtofloat(xx)
>>> print("fxx= "+fxx)
>>> print("dxx= "+dxx)
>>> end
>>>
>>> VV
>>> the result is :
>>> ^^
>>> [tlei:/lustre/scratch/tlei/hybrid-GSI/tlei-postprocessing/dr-obfits]\$
>>> ncl deb.ncl
>>> University Corporation for Atmospheric Research
>>> NCAR Command Language Version 6.0.0
>>> The use of this software is governed by a License Agreement.
>>> See http://www.ncl.ucar.edu/ for more details.
>>> (0) fxx= 2.71225
>>> (0) dxx= 2.71225
>>>
>>> VV
>>> So, it is still confusing.
>>> Thanks again.
>>> Further suggestions/comments are still appreciated deeply.
>>> Ting
>>>
>>>
>>>
>>>
>>>> Date: Wed, 31 Jul 2013 08:26:00 +0000
>>>> From: arne.melsom@met.no
>>>> To: leiting2002@hotmail.com
>>>> CC: ncl-talk@ucar.edu
>>>> Subject: Re: [ncl-talk] lost precision with string2float?
>>>>
>>>>
>>>> Hi Ting!
>>>>
>>>> You lose precision since you are storing a 12-digit number
>>>> represented as type 'float' (4-byte real), I think. If you wish to
>>>> keep the precision, you're really asking ncl to do the impossible,
>>>> since a 4-byte real has 6-9 significant digits (see
>>>> http://en.wikipedia.org/wiki/Single-precision_floating-point_format
>>>> for an explanation).
>>>>
>>>> I advise you to use stringtodouble if you need to retain full
>>>> ncl 0> str_value= "2.26410202121"
>>>> ncl 1> r4_value = stringtofloat(str_value)
>>>> ncl 2> print((/r4_value/))
>>>> (0) 2.264102
>>>> ncl 3> r8_value = stringtodouble(str_value)
>>>> ncl 4> print((/r8_value/))
>>>> (0) 2.26410202121
>>>>
>>>> As you can see, using stringtofloat my implementation gives one more
>>>> digit than in your example (which is still a round-off from the
>>>> original value). I think this could be machine-dependent, but I'm
>>>> not sure.
>>>>
>>>> Good luck!
>>>> Arne M.
>>>>
>>>>
>>>> ----- Original Message -----
>>>>> Dear NCLers,
>>>>> The lost precision in function of stringtofloat is very confusing.
>>>>> A snapshot of my script is :
>>>>> \$)G!T!T!T
>>>>> \$)A#.#.#.#.#.
>>>>> datafile(0)="/pan2/projects/hybda/tlei/hbrid-gsi/experiments/dr-processing/dr-obsfit/dr-andrew/dr-new-test/t126-ens3dvar-factq0.005-tune1_temp_24_tl3wk_date_2011072300.txtout"
>>>>> print("------------------")
>>>>> print(str_data)
>>>>> print("------------------")
>>>>>
>>>>> print("thinkyy1 "+str_get_field(str_data(2:11),10 ," "))
>>>>> datarmstr(0,:) = stringtofloat(str_get_field(str_data(2:11),10 ,"
>>>>> "))
>>>>> print("output "+ datarmstr(0,:))
>>>>> exit
>>>>> #.#.#.#.#.#.#.
>>>>>
>>>>> #V#V
>>>>>
>>>>> the output was :
>>>>> ^^
>>>>> (0) ------------------
>>>>>
>>>>>
>>>>> Variable: str_data
>>>>> Type: string
>>>>> Total Size: 184 bytes
>>>>> 23 values
>>>>> Number of Dimensions: 1
>>>>> Dimensions and sizes: [23]
>>>>> Coordinates:
>>>>> Number Of Attributes: 1
>>>>> _FillValue : missing
>>>>> (0) t126-ens3dvar-factq0.005-tune1
>>>>> (1) 24
>>>>> (2) Level: 950 GL: 8.24180969913 NH: 9.03959532222 SH:
>>>>> 4.20871527148
>>>>> TR: 2.26410202121
>>>>> (3) Level: 850 GL: 3.40322092967 NH: 3.62214877696 SH:
>>>>> 2.20873685424
>>>>> TR: 1.25765875807
>>>>> (4) Level: 750 GL: 1.64514061274 NH: 1.62367548789 SH:
>>>>> 1.89728488578
>>>>> TR: 1.76729637694
>>>>> (5) Level: 650 GL: 1.36298927422 NH: 1.33075976043 SH:
>>>>> 2.75972659747
>>>>> TR: 1.00703519724
>>>>> (6) Level: 550 GL: 1.03182087585 NH: 1.00696025671 SH:
>>>>> 1.56631219346
>>>>> TR: 1.1095806198
>>>>> (7) Level: 450 GL: 1.32992905798 NH: 1.36271631568 SH:
>>>>> 1.18470011083
>>>>> TR: 1.01202546831
>>>>> (8) Level: 350 GL: 1.21252777266 NH: 1.22452270919 SH:
>>>>> 1.4303300868
>>>>> TR: 0.812163178701
>>>>> (9) Level: 250 GL: 1.70174899943 NH: 1.72542880914 SH:
>>>>> 1.89613201475
>>>>> TR: 1.00688183094
>>>>> (10) Level: 150 GL: 2.54344055051 NH: 2.64716957083 SH:
>>>>> 2.15570650502
>>>>> TR: 1.92993841914
>>>>> (11) Level: 50 GL: 3.73654217549 NH: 3.34410145973 SH:
>>>>> 3.07788348411
>>>>> TR: 5.7543725482
>>>>> (12) tmpnumLevel: 950 GL: 16381.0 NH: 14216.0 SH: 823.0 TR: 1342.0
>>>>> (13) tmpnumLevel: 850 GL: 7427.0 NH: 6607.0 SH: 329.0 TR: 491.0
>>>>> (14) tmpnumLevel: 750 GL: 6001.0 NH: 5334.0 SH: 254.0 TR: 413.0
>>>>> (15) tmpnumLevel: 650 GL: 8412.0 NH: 7609.0 SH: 303.0 TR: 500.0
>>>>> (16) tmpnumLevel: 550 GL: 7811.0 NH: 7067.0 SH: 258.0 TR: 486.0
>>>>> (17) tmpnumLevel: 450 GL: 8011.0 NH: 7102.0 SH: 325.0 TR: 584.0
>>>>> (18) tmpnumLevel: 350 GL: 7837.0 NH: 6834.0 SH: 517.0 TR: 486.0
>>>>> (19) tmpnumLevel: 250 GL: 28293.0 NH: 26201.0 SH: 937.0 TR: 1155.0
>>>>> (20) tmpnumLevel: 150 GL: 6086.0 NH: 5067.0 SH: 441.0 TR: 578.0
>>>>> (21) tmpnumLevel: 50 GL: 8215.0 NH: 6109.0 SH: 692.0 TR: 1414.0
>>>>> (22) totnum tot: 104474.0 NH: 92146.0 SH: 4879.0 TR: 7449.0
>>>>> (0) ------------------
>>>>>
>>>>> (0) ------------------
>>>>> (0) thinkyy1 2.26410202121
>>>>> (1) thinkyy1 1.25765875807
>>>>> (2) thinkyy1 1.76729637694
>>>>> (3) thinkyy1 1.00703519724
>>>>> (4) thinkyy1 1.1095806198
>>>>> (5) thinkyy1 1.01202546831
>>>>> (6) thinkyy1 0.812163178701
>>>>> (7) thinkyy1 1.00688183094
>>>>> (8) thinkyy1 1.92993841914
>>>>> (9) thinkyy1 5.7543725482
>>>>> (0) output 2.2641
>>>>> (1) output 1.25766
>>>>> (2) output 1.7673
>>>>> (3) output 1.00704
>>>>> (4) output 1.10958
>>>>> (5) output 1.01203
>>>>> (6) output 0.812163
>>>>> (7) output 1.00688
>>>>> (8) output 1.92994
>>>>> (9) output 5.75437
>>>>>
>>>>> VV
>>>>>
>>>>> It seems,say, the string 2.26410202121 was converted to 2.2641
>>>>> with
>>>>> quite a few numbers lost.
>>>>> I tried direct converting the string, namely use
>>>>> "stringtofloat(2.2641020212), the converted result would be
>>>>> exactly
>>>>> the same. Why would such lost numbers happen in the above example?
>>>>> Best,
>>>>> Ting
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ncl-talk mailing list
>>>>> List instructions, subscriber options, unsubscribe:
>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
>
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Wed Jul 31 10:01:51 2013

This archive was generated by hypermail 2.1.8 : Thu Aug 01 2013 - 15:55:04 MDT