Re: NCL 5.1.1: Issues with concatenating a subset of an array from multiple files

From: Dennis Shea <shea_at_nyahnyahspammersnyahnyah>
Date: Wed Sep 22 2010 - 12:43:16 MDT

NCL does what it is supposed to do but the result
may not be what you expect!

There is a duplicate time. The last time of the 1st file and
the first time of the second file are the same.

===================================
snip]
(22) 2009-08-14_22:00:00 2009081422
(23) 2009-08-14_23:00:00 2009081423
(24) 2009-08-15_00:00:00 2009081500 <==

(25) 2009-08-15_00:00:00 2009081500 <==
(26) 2009-08-15_01:00:00 2009081501
(27) 2009-08-15_02:00:00 2009081502
[snip]
===================================

The netCDF operators [ncrcat] would do the same thing as NCL.

It is the user's responsibility to look at the data and
make decisions on how to handle conflicts.

The attached script shows one approach to handle the situation.

Good luck

On 09/22/2010 12:05 PM, David Brown wrote:
> Hi Yongxin,
>
> There were some fairly serious bugs in the addfiles implementation in
> NCL 5.1.0, most of which were fixed for 5.1.1. A problem that remained
> in 5.1.1 involved vector subscripting along the aggregated dimension.
> Since you are not using vector subscripting that would not affect you.
>
> However, your statements concerning the behavior of addfiles suggest
> to me that you are expecting addfiles to behave as it did prior to NCL
> 5.1.0. We considered that behavior a bug and spent considerable energy
> fixing it.
>
> Given your situation of 2 files with a variable RAIN that is
> dimensioned 25, 765, 1191 and the following statement:
>
> obs_rain = rain_fin[:]->RAIN(0:23,:,:)
>
> the resulting dimensionality should be 24,765,1191. In other words,
> the left-most dimension is implicitly aggregated into a dimension of
> 50 elements (assuming 'cat' mode). You are just picking the first 24
> of them. It is true that prior to 5.1.0 the behavior of this statement
> was different: it would have picked the first 24 elements of each file
> aggregated. But based on much user input we decided that that behavior
> was incorrect: it was inconsistent with the normal behavior of NCL's
> array subscripting syntax and it made it difficult in particular to
> work with files where the number of elements along the aggregated
> dimension varies. I hope this is clear and you can appreciate why the
> change was made. Anyway I think NCL is working as currently documented
> for you.
> -dave
>
>
> On Sep 22, 2010, at 10:53 AM, Yongxin Zhang wrote:
>
>> Hi Dennis,
>>
>> Thanks for your email. I am wondering if you have tried v5.2.1
>> with reading out and then concatenating a subset of the array (say 3
>> time steps from the first WRF file and 4 time steps from the second
>> WRF file) since that is where v5.1.0 and v5.1.1 have some issues.
>>
>> Yongxin
>>
>> On Wed, Sep 22, 2010 at 10:44 AM, Dennis Shea<shea@ucar.edu> wrote:
>> v5.1.0 was released in March 2009
>> v5.1.1 was released in June 2009
>>
>> http://www.ncl.ucar.edu/prev_releases.shtml#AddfilesBugsFixed5.1.1
>>
>> =================================
>> The current version is v5.2.1
>> It may be downloaded from: http://www.ncl.ucar.edu/Download/
>>
>> It works correctly on the following simple test.
>> =================================
>>
>> I tried v5.2.1 with 2 WRF files. One had 5 time steps;
>> the other had 6 time steps
>>
>> ===> file 1
>> netcdf wrf-00-09.sample_t0-4 {
>> dimensions:
>> Time = UNLIMITED ; // (5 currently)
>> [snip]
>> variables:
>> float PH(Time, bottom_top_stag, south_north, west_east) ;
>>
>> ===> file 2
>> netcdf wrf-00-09.sample_t5-10 {
>> dimensions:
>> Time = UNLIMITED ; // (6 currently)
>> [snip]
>> variables:
>> float PH(Time, bottom_top_stag, south_north, west_east) ;
>> [snip]
>>
>> +++++++++
>>
>>
>> fils = systemfunc("ls wrf-00-09*nc")
>> ph= f[:]->PH
>> printVarSummary(ph)
>>
>>
>> Variable: ph
>> Type: float
>> Total Size: 90700632 bytes
>> 22675158 values
>> Number of Dimensions: 4
>> Dimensions and sizes: [Time | 11] x [bottom_top_stag | 58] x
>> [south_north | 99] x [west_east | 359]
>> Coordinates:
>> Number Of Attributes: 6
>> FieldType : 104
>> MemoryOrder : XYZ
>> description : perturbation geopotential
>> units : m2 s-2
>> stagger : Z
>> coordinates : XLONG XLAT
>>
>>
>>
>> On 09/22/2010 09:22 AM, Yongxin Zhang wrote:
>> I have two netCDF files that contain only one variable, RAIN. RAIN is
>> dimensioned as (time, south_north, west_east) where time=25,
>> south_north=765 and west_east=1191. I needed to read out the first 24
>> data (i.e., time=0 to 23) at each grid point from each file and then
>> concatenate them to become a new variable. I used "systemfunc" and
>> "addfiles" but strange thing was that when I used the following
>> statement
>>
>> obs_rain = rain_fin[:]->RAIN(0:23,:,:)
>>
>> I would get obs_rain dimensioned as (24, 765, 1191) instead of (48,
>> 765,
>> 1191). Same thing happened even though I explicitly set
>> ListSetType(rain_fin,"cat"). However, if I used this statement
>>
>> obs_rain00 = rain_fin[:]->RAIN(:,:,:)
>>
>> I would get obs_rain00 dimensioned as (50, 765, 1191) which is
>> right. I
>> tried this on two different Linux clusters at NCAR using both NCL
>> 5.1.1
>> and NCL 5.1.0 and I got the same results. I am not sure if there is a
>> bug in NCL 5.1.1 and NCL 5.1.0 or I need to set something else. Please
>> advise. Thanks.
>>
>> Yongxin
>>

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk

Received on Wed Sep 22 12:43:22 2010

This archive was generated by hypermail 2.1.8 : Mon Oct 04 2010 - 08:55:55 MDT