Re: lost precision with string2float?

From: <leiting2002_at_nyahnyahspammersnyahnyah>
Date: Wed Jul 31 2013 - 06:35:41 MDT

Arne,
Sorry for my stupid mistake:). With the corrected one:
^^
[tlei:/lustre/scratch/tlei/hybrid-GSI/tlei-postprocessing/dr-obfits]$ cat deb.ncl

load "/sw/xt-cle3.1/ncl/5.0.0/sles11.1_pgi11.6.0/lib/ncarg/nclscripts/csm/gsn_code.ncl"
load "/sw/xt-cle3.1/ncl/5.0.0/sles11.1_pgi11.6.0/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
load "/sw/xt-cle3.1/ncl/5.0.0/sles11.1_pgi11.6.0/lib/ncarg/nclscripts/csm/contributed.ncl"
load "/sw/xt-cle3.1/ncl/5.0.0/sles11.1_pgi11.6.0/lib/ncarg/nclscripts/csm/shea_util.ncl"
begin
xx="2.712245356"
fxx=stringtofloat(xx)
dxx=stringtodouble(xx)
print("fxx= "+fxx)
print("dxx= "+dxx)
end
Vv
It gave the same result.
Thank you .
Ting

> Date: Wed, 31 Jul 2013 12:22:23 +0000
> From: arne.melsom@met.no
> To: leiting2002@hotmail.com
> CC: ncl-talk@ucar.edu
> Subject: Re: [ncl-talk] lost precision with string2float?
>
> Hi,
> I don't know why you get the same result with stringtofloat and stringtodouble. But I note that your 'Kraken' example has an error
> xx="2.712245356"
> fxx=stringtofloat(xx)
> dxx=stringtofloat(xx)
> Here, you use stringtofloat for both fxx and dxx. If there really is a problem with stringtodouble, I hope that someone else can help you; I don't know what may be wrong. But I can assure that, with 12 digits, stringtofloat will always give you a round-off error.
> Best regards,
> Arne
>
> ----- Original Message -----
> > Hi, Arne,
> > Thank you so much for your suggestions.
> > I had suspected on the impact of the difference between stringtofloat
> > and stringtodouble and found no differences. Just now, I just tried it
> > again. seems the same. In the previous example, I changed it into :
> > ^^
> > fxx = stringtofloat(str_get_field(str_data(2:11),10 ," "))
> > dxx = stringtodouble(str_get_field(str_data(2:11),10 ," "))
> > print("output fxx "+ fxx)
> > print("output dxx "+ dxx)
> > VV
> > It gave the same results. In case it is machine dependent, I tried it
> > on another machine (Kraken).
> > The test ncl file is
> > [tlei:/lustre/scratch/tlei/hybrid-GSI/tlei-postprocessing/dr-obfits]$
> > cat deb.ncl
> > ^^
> > load
> > "/sw/xt-cle3.1/ncl/5.0.0/sles11.1_pgi11.6.0/lib/ncarg/nclscripts/csm/gsn_code.ncl"
> > load
> > "/sw/xt-cle3.1/ncl/5.0.0/sles11.1_pgi11.6.0/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
> > load
> > "/sw/xt-cle3.1/ncl/5.0.0/sles11.1_pgi11.6.0/lib/ncarg/nclscripts/csm/contributed.ncl"
> > load
> > "/sw/xt-cle3.1/ncl/5.0.0/sles11.1_pgi11.6.0/lib/ncarg/nclscripts/csm/shea_util.ncl"
> > begin
> > xx="2.712245356"
> > fxx=stringtofloat(xx)
> > dxx=stringtofloat(xx)
> > print("fxx= "+fxx)
> > print("dxx= "+dxx)
> > end
> >
> > VV
> > the result is :
> > ^^
> > [tlei:/lustre/scratch/tlei/hybrid-GSI/tlei-postprocessing/dr-obfits]$
> > ncl deb.ncl
> > Copyright (C) 1995-2011 - All Rights Reserved
> > University Corporation for Atmospheric Research
> > NCAR Command Language Version 6.0.0
> > The use of this software is governed by a License Agreement.
> > See http://www.ncl.ucar.edu/ for more details.
> > (0) fxx= 2.71225
> > (0) dxx= 2.71225
> >
> > VV
> > So, it is still confusing.
> > Thanks again.
> > Further suggestions/comments are still appreciated deeply.
> > Ting
> >
> >
> >
> >
> > > Date: Wed, 31 Jul 2013 08:26:00 +0000
> > > From: arne.melsom@met.no
> > > To: leiting2002@hotmail.com
> > > CC: ncl-talk@ucar.edu
> > > Subject: Re: [ncl-talk] lost precision with string2float?
> > >
> > >
> > > Hi Ting!
> > >
> > > You lose precision since you are storing a 12-digit number
> > > represented as type 'float' (4-byte real), I think. If you wish to
> > > keep the precision, you're really asking ncl to do the impossible,
> > > since a 4-byte real has 6-9 significant digits (see
> > > http://en.wikipedia.org/wiki/Single-precision_floating-point_format
> > > for an explanation).
> > >
> > > I advise you to use stringtodouble if you need to retain full
> > > precision in your example:
> > > ncl 0> str_value= "2.26410202121"
> > > ncl 1> r4_value = stringtofloat(str_value)
> > > ncl 2> print((/r4_value/))
> > > (0) 2.264102
> > > ncl 3> r8_value = stringtodouble(str_value)
> > > ncl 4> print((/r8_value/))
> > > (0) 2.26410202121
> > >
> > > As you can see, using stringtofloat my implementation gives one more
> > > digit than in your example (which is still a round-off from the
> > > original value). I think this could be machine-dependent, but I'm
> > > not sure.
> > >
> > > Good luck!
> > > Arne M.
> > >
> > >
> > > ----- Original Message -----
> > > > Dear NCLers,
> > > > The lost precision in function of stringtofloat is very confusing.
> > > > A snapshot of my script is :
> > > >
> > > >
> > > > datafile(0)="/pan2/projects/hybda/tlei/hbrid-gsi/experiments/dr-processing/dr-obsfit/dr-andrew/dr-new-test/t126-ens3dvar-factq0.005-tune1_temp_24_tl3wk_date_2011072300.txtout"
> > > > str_data = asciiread(datafile(0),-1,"string")
> > > > print("------------------")
> > > > print(str_data)
> > > > print("------------------")
> > > >
> > > > print("thinkyy1 "+str_get_field(str_data(2:11),10 ," "))
> > > > datarmstr(0,:) = stringtofloat(str_get_field(str_data(2:11),10 ,"
> > > > "))
> > > > print("output "+ datarmstr(0,:))
> > > > exit
> > > >
> > > >
> > > > ֣
> > > >
> > > > the output was :
> > > > ^^
> > > > (0) ------------------
> > > >
> > > >
> > > > Variable: str_data
> > > > Type: string
> > > > Total Size: 184 bytes
> > > > 23 values
> > > > Number of Dimensions: 1
> > > > Dimensions and sizes: [23]
> > > > Coordinates:
> > > > Number Of Attributes: 1
> > > > _FillValue : missing
> > > > (0) t126-ens3dvar-factq0.005-tune1
> > > > (1) 24
> > > > (2) Level: 950 GL: 8.24180969913 NH: 9.03959532222 SH:
> > > > 4.20871527148
> > > > TR: 2.26410202121
> > > > (3) Level: 850 GL: 3.40322092967 NH: 3.62214877696 SH:
> > > > 2.20873685424
> > > > TR: 1.25765875807
> > > > (4) Level: 750 GL: 1.64514061274 NH: 1.62367548789 SH:
> > > > 1.89728488578
> > > > TR: 1.76729637694
> > > > (5) Level: 650 GL: 1.36298927422 NH: 1.33075976043 SH:
> > > > 2.75972659747
> > > > TR: 1.00703519724
> > > > (6) Level: 550 GL: 1.03182087585 NH: 1.00696025671 SH:
> > > > 1.56631219346
> > > > TR: 1.1095806198
> > > > (7) Level: 450 GL: 1.32992905798 NH: 1.36271631568 SH:
> > > > 1.18470011083
> > > > TR: 1.01202546831
> > > > (8) Level: 350 GL: 1.21252777266 NH: 1.22452270919 SH:
> > > > 1.4303300868
> > > > TR: 0.812163178701
> > > > (9) Level: 250 GL: 1.70174899943 NH: 1.72542880914 SH:
> > > > 1.89613201475
> > > > TR: 1.00688183094
> > > > (10) Level: 150 GL: 2.54344055051 NH: 2.64716957083 SH:
> > > > 2.15570650502
> > > > TR: 1.92993841914
> > > > (11) Level: 50 GL: 3.73654217549 NH: 3.34410145973 SH:
> > > > 3.07788348411
> > > > TR: 5.7543725482
> > > > (12) tmpnumLevel: 950 GL: 16381.0 NH: 14216.0 SH: 823.0 TR: 1342.0
> > > > (13) tmpnumLevel: 850 GL: 7427.0 NH: 6607.0 SH: 329.0 TR: 491.0
> > > > (14) tmpnumLevel: 750 GL: 6001.0 NH: 5334.0 SH: 254.0 TR: 413.0
> > > > (15) tmpnumLevel: 650 GL: 8412.0 NH: 7609.0 SH: 303.0 TR: 500.0
> > > > (16) tmpnumLevel: 550 GL: 7811.0 NH: 7067.0 SH: 258.0 TR: 486.0
> > > > (17) tmpnumLevel: 450 GL: 8011.0 NH: 7102.0 SH: 325.0 TR: 584.0
> > > > (18) tmpnumLevel: 350 GL: 7837.0 NH: 6834.0 SH: 517.0 TR: 486.0
> > > > (19) tmpnumLevel: 250 GL: 28293.0 NH: 26201.0 SH: 937.0 TR: 1155.0
> > > > (20) tmpnumLevel: 150 GL: 6086.0 NH: 5067.0 SH: 441.0 TR: 578.0
> > > > (21) tmpnumLevel: 50 GL: 8215.0 NH: 6109.0 SH: 692.0 TR: 1414.0
> > > > (22) totnum tot: 104474.0 NH: 92146.0 SH: 4879.0 TR: 7449.0
> > > > (0) ------------------
> > > >
> > > > (0) ------------------
> > > > (0) thinkyy1 2.26410202121
> > > > (1) thinkyy1 1.25765875807
> > > > (2) thinkyy1 1.76729637694
> > > > (3) thinkyy1 1.00703519724
> > > > (4) thinkyy1 1.1095806198
> > > > (5) thinkyy1 1.01202546831
> > > > (6) thinkyy1 0.812163178701
> > > > (7) thinkyy1 1.00688183094
> > > > (8) thinkyy1 1.92993841914
> > > > (9) thinkyy1 5.7543725482
> > > > (0) output 2.2641
> > > > (1) output 1.25766
> > > > (2) output 1.7673
> > > > (3) output 1.00704
> > > > (4) output 1.10958
> > > > (5) output 1.01203
> > > > (6) output 0.812163
> > > > (7) output 1.00688
> > > > (8) output 1.92994
> > > > (9) output 5.75437
> > > >
> > > > VV
> > > >
> > > > It seems,say, the string 2.26410202121 was converted to 2.2641
> > > > with
> > > > quite a few numbers lost.
> > > > I tried direct converting the string, namely use
> > > > "stringtofloat(2.2641020212), the converted result would be
> > > > exactly
> > > > the same. Why would such lost numbers happen in the above example?
> > > > Your help is appreciated.
> > > > Best,
> > > > Ting
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > ncl-talk mailing list
> > > > List instructions, subscriber options, unsubscribe:
> > > > http://mailman.ucar.edu/mailman/listinfo/ncl-talk
                                               

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Wed Jul 31 06:35:53 2013

This archive was generated by hypermail 2.1.8 : Thu Aug 01 2013 - 15:55:04 MDT