From: James Scott <james.d.scott_at_nyahnyahspammersnyahnyah>

Date: Mon Oct 17 2011 - 12:26:19 MDT

Date: Mon Oct 17 2011 - 12:26:19 MDT

Chuck,

I think the problem has something to do with the avg function doing floating

point division rather than integer division.

If you use the sum function and divide by the integer 60 you get the correct

result:

ncl 14> print(sum(tmp)/60)

(0) 20060109

If you use the avg function you get a floating point result:

ncl 15> print(avg(tmp))

(0) 2.006011e+07

Similarly, if you use the sum function and divide by floating point 60.0,

you get a floating point result.

ncl 16> print(sum(tmp)/60.)

(0) 2.006011e+07

-Jamie

--------------------------------------------------------------------

Message: 5

Date: Mon, 17 Oct 2011 08:53:31 -0600

From: Charles Bardeen <bardeenc@ucar.edu>

Subject: Re: Problem with averaging integers?

To: Dennis Shea <shea@ucar.edu>

Cc: "ncl-talk@ucar.edu" <ncl-talk@ucar.edu>

Message-ID: <1DFCCC94-60D0-4F2D-BB11-280A69554056@ucar.edu>

Content-Type: text/plain; charset=utf-8

Hi Dennis,

I am sorry I wasn't clear. I am not concerned about how the float is

printed. I am concerned that the number ends in 108 rather than 109. It was

set to be 20060109 and the average is 20060108. That appears to be true with

your result too.

Thanks,

Chuck

On Oct 17, 2011, at 8:18 AM, Dennis Shea <shea@ucar.edu> wrote:

*> There is nothing wrong. The 'print' value is
*

*> at 6 decimal values which is generally 'sufficient' for floats.
*

*> printing the full value ... gives the expected anser.
*

*>
*

*>
*

*> tmp = new((/ 60 /), integer)
*

*> tmp = 20060109 ; do not use tmp(:) [slower]
*

*> avg_tmp8 = avg(tmp)
*

*> print(avg_tmp8)
*

*> print(sprintf("%20.13f", avg_tmp8))
*

*>
*

*> ===
*

*> Variable: avg_tmp8
*

*> Type: float
*

*> Total Size: 4 bytes
*

*> 1 values
*

*> Number of Dimensions: 1
*

*> Dimensions and sizes: [1]
*

*> Coordinates:
*

*> Number Of Attributes: 1
*

*> _FillValue : -2.147484e+09
*

*> (0) 2.006011e+07
*

*>
*

*>
*

*> (0) 20060108.0000000000000 <=== sprintf
*

*>
*

*>
*

*> On 10/16/11 3:52 PM, Charles Bardeen wrote:
*

*>> The following code yields an average that is 1 from the true average ?
*

*>>
*

*>>
*

*>> Copyright (C) 1995-2010 - All Rights Reserved
*

*>> University Corporation for Atmospheric Research
*

*>> NCAR Command Language Version 5.2.1
*

*>> The use of this software is governed by a License Agreement.
*

*>> See http://www.ncl.ucar.edu/ for more details.
*

*>> ncl 0> tmp = new((/ 60 /), integer)
*

*>> ncl 1> tmp(:) = 20060109
*

*>> ncl 2> print(avg(tmp))
*

*>> (0) 2.006011e+07
*

*>> ncl 3> print(doubletointeger(avg(tmp)))
*

*>> (0) 20060108
*

*>> ncl 4>
*

*>>
*

*>> Any idea why NCL is making this mistake. Is it an overflow or rounding
*

*>> problem? If I use smaller numbers, then it works correctly ?
*

*>>
*

*>> ncl 8> tmp(:) = 2006
*

*>> ncl 9> print(avg(tmp))
*

*>> (0) 2006
*

*>> ncl 10> print(doubletointeger(avg(tmp)))
*

*>> (0) 2006
*

*>> ncl 11>
*

*>>
*

*>>
*

*>> Thanks,
*

*>>
*

*>> Chuck
*

*>>
*

*>>
*

*>> --------------------------------------------------------------
*

*>> Charles Bardeen
*

*>> National Center for Atmospheric Research
*

*>> P.O. Box 3000, Boulder, CO 80307-3000
*

*>> Phone: (303) 497-1752, Fax: (303) 497-1400
*

*>> bardeenc@ucar.edu <mailto:bardeenc@ucar.edu>
*

_______________________________________________

ncl-talk mailing list

List instructions, subscriber options, unsubscribe:

http://mailman.ucar.edu/mailman/listinfo/ncl-talk

Received on Mon Oct 17 12:26:52 2011

*
This archive was generated by hypermail 2.1.8
: Wed Oct 19 2011 - 13:36:10 MDT
*