Re: Unlimited Time Coord

Hi Carl,

Likely, an NCL developer will have to answer for a definitive answer.
---------------------------------------------------------------------
However, to my knowledge,
NCL (like all other tools) uses the standard Unidata software
to read data from a netCDF file. Unidata states that the nc sofware
was developed for robustness

The following is conceptual:

When there is no unlimited dimension, the data are written as blocks.
When importing data, this allows the software to access
(seek and read) individual variables in the file as one
'big block'. Only one disk access (seek & read) is used.

When there is an unlimited dimension, the variables are interleaved.
To access data from multiple times requires multiple seeks and reads.
If the files are large, this can be time consuming.

==============================

Let's say there are 2 variables a(time,lev,lat,lon), b(time,lat,lon)
where time>1

If the record dimension is not tagged as unlimited ,
the data are written and read as a *block write*.

      ---------
      netCDF header stuff:
      variable/dimension names, dimension sizes, attributes,......
      ---------
      |_______| t=0 [lev,lat,lon]
      |_______| t=1
      |_______|
      |__ a___|
      |_______|
      |_______|
      |_______| t=N

followed by

      ---------
      |_______| t=0 [lat,lon]
      |_______| t=1
      |_______|
      |__ b___|
      |_______|
      |_______|
      |_______| t=N

With an unlimited dimension the variables are *interleaved*
with the record dimension [typically, "time"]

     |___ a___| a(lev,lat,lon) at t=0
     |___ b___| b(lat,lon) at t=0

     |___ a___| a(lev,lat,lon) at t=1
     |___ b___| b(lat,lon) at t=2

          : :

     |___ a___| a(lev,lat,lon) at t=N
     |___ b___| b(lat,lon) at t=N

========================================================
As I am sure you are aware, if you are on a multiuser syatem with
lots of IO traffic and competition for memory, the external
environment can significantly degrade performance.

Just yesterday, a script took 2 min to complete on one VERY busy
system and less than 10 seconds on another system. Mainly
cuz the data traffic on the network. :-(

========================================================
pNetCDF, which NCL does not use, could be used to speed
data access.
=======================================================

D

On 02/01/2013 09:07 AM, Carl Schreck wrote:
> In one of the earlier releases of NCL, it took quite a bit longer to
> open files that had time as an unlimited time coordinate than files that
> didn't. Can anyone confirm whether that has changed in more recent releases?
>
> Thanks!
> Carl
>
> --
> Carl J. Schreck III, PhD
> Cooperative Institute for Climate and Satellites (CICS-NC)
> North Carolina State University | NOAA's National Climatic Data Center
> Email: carl@cicsnc.org <mailto:carl@cicsnc.org> | Tel: 828-257-3140
> http://www.cyclonecenter.org/ | http://monitor.cicsnc.org/mjo/
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Fri Feb 1 10:42:33 2013