Re: irregular values from fspan

From: Bormann, Kathryn J (3246-Affiliate) <Kathryn.J.Bormann_at_nyahnyahspammersnyahnyah>
Date: Tue Nov 26 2013 - 12:55:26 MST

Thanks Dave – that's it! Switching to double resolves the issue.

Kat

From: David Brown <dbrown@ucar.edu<mailto:dbrown@ucar.edu>>
Date: Tuesday, November 26, 2013 11:44 AM
To: kat b <kathryn.j.bormann@jpl.nasa.gov<mailto:kathryn.j.bormann@jpl.nasa.gov>>
Cc: "ncl-talk@ucar.edu<mailto:ncl-talk@ucar.edu>" <ncl-talk@ucar.edu<mailto:ncl-talk@ucar.edu>>
Subject: Re: irregular values from fspan

Hi Kathryn,
You are just running into an issue of the print precision. By default, NCL prints float values with a precision of 6 significant digits, causing in your case truncation of the decimal part of the value.
If you change the print statement to:

print(sprintf("%10.2f",seq(0:6))) ;print test of sequence from span

You will see the full precision of your values:

(0) 4229667.00
(1) 4229665.50
(2) 4229664.00
(3) 4229662.50
(4) 4229661.00
(5) 4229659.50
(6) 4229658.00

Note however that you are very near the maximum possible float precision, so in some case you may see float point rounding inaccuracies. You could switch to double precision if you have trouble.
 -dave

On Nov 26, 2013, at 12:25 PM, "Bormann, Kathryn J (3246-Affiliate)" <Kathryn.J.Bormann@jpl.nasa.gov<mailto:Kathryn.J.Bormann@jpl.nasa.gov>> wrote:

y1 = 4229667.
y2 = 4179327.
printVarSummary(y1)

Variable: y1
Type: float
Total Size: 4 bytes
            1 values
Number of Dimensions: 1
Dimensions and sizes: [1]
Coordinates:
ncl 25> printVarSummary(y2)

Variable: y2
Type: float
Total Size: 4 bytes
            1 values
Number of Dimensions: 1
Dimensions and sizes: [1]
Coordinates:
n = 33560
seq = fspan( y1, y2, n)
print(seq(0:6)) ;print test of sequence from fspan

Variable: seq (subsection)
Type: float
Total Size: 28 bytes
            7 values
Number of Dimensions: 1
Dimensions and sizes: [7]
Coordinates:
(0) 4229667
(1) 4229666
(2) 4229664
(3) 4229662
(4) 4229661
(5) 4229660
(6) 4229658
ncl 29> spacing = (y2 - y1)/n
ncl 30> print(spacing)

;---expected spacing
Variable: spacing
Type: float
Total Size: 4 bytes
            1 values
Number of Dimensions: 1
Dimensions and sizes: [1]
Coordinates:
(0) -1.5

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Tue Nov 26 12:55:40 2013

This archive was generated by hypermail 2.1.8 : Wed Dec 04 2013 - 20:42:38 MST