Re: Unlimited Time Coord

Dennis provides a succinct description of the difference between a file with and without an unlimited dimension.

But there is another factor that can slow down NCL's access to a file with an unlimited dimension:
NCL reads the coordinate variable values for the unlimited dimension and caches these values.
This requires actually reading a single value from each record, which can be expensive for a file with a large unlimited
dimension. Using the example below these are the values of t at t(0) -- t(N). Since there is no requirement that there
be even spacing between the coordinate values there is no foolproof way to get all the values without looking
at each record at least once. We could (and maybe should) implement an option that would allow the user to
declare that the data is evenly spaced along the unlimited dimension. This would induce NCL to look at just a
few records and infer the rest of the coordinate values.
 -dave

On Feb 1, 2013, at 10:42 AM, Dennis Shea wrote:

> Hi Carl,
>
> Likely, an NCL developer will have to answer for a definitive answer.
> ---------------------------------------------------------------------
> However, to my knowledge,
> NCL (like all other tools) uses the standard Unidata software
> to read data from a netCDF file. Unidata states that the nc sofware
> was developed for robustness
>
> The following is conceptual:
>
> When there is no unlimited dimension, the data are written as blocks.
> When importing data, this allows the software to access
> (seek and read) individual variables in the file as one
> 'big block'. Only one disk access (seek & read) is used.
>
> When there is an unlimited dimension, the variables are interleaved.
> To access data from multiple times requires multiple seeks and reads.
> If the files are large, this can be time consuming.
>
> ==============================
>
> Let's say there are 2 variables a(time,lev,lat,lon), b(time,lat,lon)
> where time>1
>
> If the record dimension is not tagged as unlimited ,
> the data are written and read as a *block write*.
>
> ---------
> netCDF header stuff:
> variable/dimension names, dimension sizes, attributes,......
> ---------
> |_______| t=0 [lev,lat,lon]
> |_______| t=1
> |_______|
> |__ a___|
> |_______|
> |_______|
> |_______| t=N
>
> followed by
>
> ---------
> |_______| t=0 [lat,lon]
> |_______| t=1
> |_______|
> |__ b___|
> |_______|
> |_______|
> |_______| t=N
>
> With an unlimited dimension the variables are *interleaved*
> with the record dimension [typically, "time"]
>
> |___ a___| a(lev,lat,lon) at t=0
> |___ b___| b(lat,lon) at t=0
>
> |___ a___| a(lev,lat,lon) at t=1
> |___ b___| b(lat,lon) at t=2
>
> : :
>
> |___ a___| a(lev,lat,lon) at t=N
> |___ b___| b(lat,lon) at t=N
>
> ========================================================
> As I am sure you are aware, if you are on a multiuser syatem with
> lots of IO traffic and competition for memory, the external
> environment can significantly degrade performance.
>
> Just yesterday, a script took 2 min to complete on one VERY busy
> system and less than 10 seconds on another system. Mainly
> cuz the data traffic on the network. :-(
>
> ========================================================
> pNetCDF, which NCL does not use, could be used to speed
> data access.
> =======================================================
>
> D
>
>
> On 02/01/2013 09:07 AM, Carl Schreck wrote:
>> In one of the earlier releases of NCL, it took quite a bit longer to
>> open files that had time as an unlimited time coordinate than files that
>> didn't. Can anyone confirm whether that has changed in more recent releases?
>>
>> Thanks!
>> Carl
>>
>> --
>> Carl J. Schreck III, PhD
>> Cooperative Institute for Climate and Satellites (CICS-NC)
>> North Carolina State University | NOAA's National Climatic Data Center
>> Email: carl@cicsnc.org <mailto:carl@cicsnc.org> | Tel: 828-257-3140
>> http://www.cyclonecenter.org/ | http://monitor.cicsnc.org/mjo/
>>
>>
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Fri Feb 1 12:06:23 2013