Re: lost precision with string2float?

From: Arne Melsom <arne.melsom_at_nyahnyahspammersnyahnyah>
Date: Wed Jul 31 2013 - 02:26:00 MDT

Hi Ting!

You lose precision since you are storing a 12-digit number represented as type 'float' (4-byte real), I think. If you wish to keep the precision, you're really asking ncl to do the impossible, since a 4-byte real has 6-9 significant digits (see http://en.wikipedia.org/wiki/Single-precision_floating-point_format for an explanation).

I advise you to use stringtodouble if you need to retain full precision in your example:
ncl 0> str_value= "2.26410202121"
ncl 1> r4_value = stringtofloat(str_value)
ncl 2> print((/r4_value/))
(0) 2.264102
ncl 3> r8_value = stringtodouble(str_value)
ncl 4> print((/r8_value/))
(0) 2.26410202121

As you can see, using stringtofloat my implementation gives one more digit than in your example (which is still a round-off from the original value). I think this could be machine-dependent, but I'm not sure.

Good luck!
Arne M.

----- Original Message -----
> Dear NCLers,
> The lost precision in function of stringtofloat is very confusing.
> A snapshot of my script is :
> ︿︿︿
> .....
> datafile(0)="/pan2/projects/hybda/tlei/hbrid-gsi/experiments/dr-processing/dr-obsfit/dr-andrew/dr-new-test/t126-ens3dvar-factq0.005-tune1_temp_24_tl3wk_date_2011072300.txtout"
> str_data = asciiread(datafile(0),-1,"string")
> print("------------------")
> print(str_data)
> print("------------------")
>
> print("thinkyy1 "+str_get_field(str_data(2:11),10 ," "))
> datarmstr(0,:) = stringtofloat(str_get_field(str_data(2:11),10 ," "))
> print("output "+ datarmstr(0,:))
> exit
> .......
>
> VV
>
> the output was :
> ^^
> (0) ------------------
>
>
> Variable: str_data
> Type: string
> Total Size: 184 bytes
> 23 values
> Number of Dimensions: 1
> Dimensions and sizes: [23]
> Coordinates:
> Number Of Attributes: 1
> _FillValue : missing
> (0) t126-ens3dvar-factq0.005-tune1
> (1) 24
> (2) Level: 950 GL: 8.24180969913 NH: 9.03959532222 SH: 4.20871527148
> TR: 2.26410202121
> (3) Level: 850 GL: 3.40322092967 NH: 3.62214877696 SH: 2.20873685424
> TR: 1.25765875807
> (4) Level: 750 GL: 1.64514061274 NH: 1.62367548789 SH: 1.89728488578
> TR: 1.76729637694
> (5) Level: 650 GL: 1.36298927422 NH: 1.33075976043 SH: 2.75972659747
> TR: 1.00703519724
> (6) Level: 550 GL: 1.03182087585 NH: 1.00696025671 SH: 1.56631219346
> TR: 1.1095806198
> (7) Level: 450 GL: 1.32992905798 NH: 1.36271631568 SH: 1.18470011083
> TR: 1.01202546831
> (8) Level: 350 GL: 1.21252777266 NH: 1.22452270919 SH: 1.4303300868
> TR: 0.812163178701
> (9) Level: 250 GL: 1.70174899943 NH: 1.72542880914 SH: 1.89613201475
> TR: 1.00688183094
> (10) Level: 150 GL: 2.54344055051 NH: 2.64716957083 SH: 2.15570650502
> TR: 1.92993841914
> (11) Level: 50 GL: 3.73654217549 NH: 3.34410145973 SH: 3.07788348411
> TR: 5.7543725482
> (12) tmpnumLevel: 950 GL: 16381.0 NH: 14216.0 SH: 823.0 TR: 1342.0
> (13) tmpnumLevel: 850 GL: 7427.0 NH: 6607.0 SH: 329.0 TR: 491.0
> (14) tmpnumLevel: 750 GL: 6001.0 NH: 5334.0 SH: 254.0 TR: 413.0
> (15) tmpnumLevel: 650 GL: 8412.0 NH: 7609.0 SH: 303.0 TR: 500.0
> (16) tmpnumLevel: 550 GL: 7811.0 NH: 7067.0 SH: 258.0 TR: 486.0
> (17) tmpnumLevel: 450 GL: 8011.0 NH: 7102.0 SH: 325.0 TR: 584.0
> (18) tmpnumLevel: 350 GL: 7837.0 NH: 6834.0 SH: 517.0 TR: 486.0
> (19) tmpnumLevel: 250 GL: 28293.0 NH: 26201.0 SH: 937.0 TR: 1155.0
> (20) tmpnumLevel: 150 GL: 6086.0 NH: 5067.0 SH: 441.0 TR: 578.0
> (21) tmpnumLevel: 50 GL: 8215.0 NH: 6109.0 SH: 692.0 TR: 1414.0
> (22) totnum tot: 104474.0 NH: 92146.0 SH: 4879.0 TR: 7449.0
> (0) ------------------
>
> (0) ------------------
> (0) thinkyy1 2.26410202121
> (1) thinkyy1 1.25765875807
> (2) thinkyy1 1.76729637694
> (3) thinkyy1 1.00703519724
> (4) thinkyy1 1.1095806198
> (5) thinkyy1 1.01202546831
> (6) thinkyy1 0.812163178701
> (7) thinkyy1 1.00688183094
> (8) thinkyy1 1.92993841914
> (9) thinkyy1 5.7543725482
> (0) output 2.2641
> (1) output 1.25766
> (2) output 1.7673
> (3) output 1.00704
> (4) output 1.10958
> (5) output 1.01203
> (6) output 0.812163
> (7) output 1.00688
> (8) output 1.92994
> (9) output 5.75437
>
> VV
>
> It seems,say, the string 2.26410202121 was converted to 2.2641 with
> quite a few numbers lost.
> I tried direct converting the string, namely use
> "stringtofloat(2.2641020212), the converted result would be exactly
> the same. Why would such lost numbers happen in the above example?
> Your help is appreciated.
> Best,
> Ting
>
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Wed Jul 31 02:26:13 2013

This archive was generated by hypermail 2.1.8 : Thu Aug 01 2013 - 15:55:04 MDT