Re: Problem with averaging integers?

Hi Dennis,

I am sorry I wasn't clear. I am not concerned about how the float is printed. I am concerned that the number ends in 108 rather than 109. It was set to be 20060109 and the average is 20060108. That appears to be true with your result too.

Thanks,

Chuck

On Oct 17, 2011, at 8:18 AM, Dennis Shea <shea@ucar.edu> wrote:

> There is nothing wrong. The 'print' value is
> at 6 decimal values which is generally 'sufficient' for floats.
> printing the full value ... gives the expected anser.
>
>
> tmp = new((/ 60 /), integer)
> tmp = 20060109 ; do not use tmp(:) [slower]
> avg_tmp8 = avg(tmp)
> print(avg_tmp8)
> print(sprintf("%20.13f", avg_tmp8))
>
> ===
> Variable: avg_tmp8
> Type: float
> Total Size: 4 bytes
> 1 values
> Number of Dimensions: 1
> Dimensions and sizes: [1]
> Coordinates:
> Number Of Attributes: 1
> _FillValue : -2.147484e+09
> (0) 2.006011e+07
>
>
> (0) 20060108.0000000000000 <=== sprintf
>
>
> On 10/16/11 3:52 PM, Charles Bardeen wrote:
>> The following code yields an average that is 1 from the true average …
>>
>>
>> Copyright (C) 1995-2010 - All Rights Reserved
>> University Corporation for Atmospheric Research
>> NCAR Command Language Version 5.2.1
>> The use of this software is governed by a License Agreement.
>> See http://www.ncl.ucar.edu/ for more details.
>> ncl 0> tmp = new((/ 60 /), integer)
>> ncl 1> tmp(:) = 20060109
>> ncl 2> print(avg(tmp))
>> (0) 2.006011e+07
>> ncl 3> print(doubletointeger(avg(tmp)))
>> (0) 20060108
>> ncl 4>
>>
>> Any idea why NCL is making this mistake. Is it an overflow or rounding
>> problem? If I use smaller numbers, then it works correctly …
>>
>> ncl 8> tmp(:) = 2006
>> ncl 9> print(avg(tmp))
>> (0) 2006
>> ncl 10> print(doubletointeger(avg(tmp)))
>> (0) 2006
>> ncl 11>
>>
>>
>> Thanks,
>>
>> Chuck
>>
>>
>> --------------------------------------------------------------
>> Charles Bardeen
>> National Center for Atmospheric Research
>> P.O. Box 3000, Boulder, CO 80307-3000
>> Phone: (303) 497-1752, Fax: (303) 497-1400
>> bardeenc@ucar.edu <mailto:bardeenc@ucar.edu>
>>
>>
>>
>>
>>
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Mon Oct 17 08:53:39 2011