Re: problem averaging integers

Chuck,

I think the problem has something to do with the avg function doing floating
point division rather than integer division.

If you use the sum function and divide by the integer 60 you get the correct
result:
ncl 14> print(sum(tmp)/60)
(0) 20060109

If you use the avg function you get a floating point result:
ncl 15> print(avg(tmp))
(0) 2.006011e+07

Similarly, if you use the sum function and divide by floating point 60.0,
you get a floating point result.
ncl 16> print(sum(tmp)/60.)
(0) 2.006011e+07

-Jamie
--------------------------------------------------------------------

Message: 5
Date: Mon, 17 Oct 2011 08:53:31 -0600
From: Charles Bardeen <bardeenc@ucar.edu>
Subject: Re: Problem with averaging integers?
To: Dennis Shea <shea@ucar.edu>
Cc: "ncl-talk@ucar.edu" <ncl-talk@ucar.edu>
Message-ID: <1DFCCC94-60D0-4F2D-BB11-280A69554056@ucar.edu>
Content-Type: text/plain; charset=utf-8
Hi Dennis,
I am sorry I wasn't clear. I am not concerned about how the float is
printed. I am concerned that the number ends in 108 rather than 109. It was
set to be 20060109 and the average is 20060108. That appears to be true with
your result too.
Thanks,
Chuck
On Oct 17, 2011, at 8:18 AM, Dennis Shea <shea@ucar.edu> wrote:
> There is nothing wrong. The 'print' value is
> at 6 decimal values which is generally 'sufficient' for floats.
> printing the full value ... gives the expected anser.
>
>
> tmp = new((/ 60 /), integer)
> tmp = 20060109 ; do not use tmp(:) [slower]
> avg_tmp8 = avg(tmp)
> print(avg_tmp8)
> print(sprintf("%20.13f", avg_tmp8))
>
> ===
> Variable: avg_tmp8
> Type: float
> Total Size: 4 bytes
> 1 values
> Number of Dimensions: 1
> Dimensions and sizes: [1]
> Coordinates:
> Number Of Attributes: 1
> _FillValue : -2.147484e+09
> (0) 2.006011e+07
>
>
> (0) 20060108.0000000000000 <=== sprintf
>
>
> On 10/16/11 3:52 PM, Charles Bardeen wrote:
>> The following code yields an average that is 1 from the true average ?
>>
>>
>> Copyright (C) 1995-2010 - All Rights Reserved
>> University Corporation for Atmospheric Research
>> NCAR Command Language Version 5.2.1
>> The use of this software is governed by a License Agreement.
>> See http://www.ncl.ucar.edu/ for more details.
>> ncl 0> tmp = new((/ 60 /), integer)
>> ncl 1> tmp(:) = 20060109
>> ncl 2> print(avg(tmp))
>> (0) 2.006011e+07
>> ncl 3> print(doubletointeger(avg(tmp)))
>> (0) 20060108
>> ncl 4>
>>
>> Any idea why NCL is making this mistake. Is it an overflow or rounding
>> problem? If I use smaller numbers, then it works correctly ?
>>
>> ncl 8> tmp(:) = 2006
>> ncl 9> print(avg(tmp))
>> (0) 2006
>> ncl 10> print(doubletointeger(avg(tmp)))
>> (0) 2006
>> ncl 11>
>>
>>
>> Thanks,
>>
>> Chuck
>>
>>
>> --------------------------------------------------------------
>> Charles Bardeen
>> National Center for Atmospheric Research
>> P.O. Box 3000, Boulder, CO 80307-3000
>> Phone: (303) 497-1752, Fax: (303) 497-1400
>> bardeenc@ucar.edu <mailto:bardeenc@ucar.edu>

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Mon Oct 17 12:26:52 2011