Re: reading variable attributes

From: Dave Allured <dave.allured_at_nyahnyahspammersnyahnyah>
Date: Wed Jul 18 2012 - 16:10:20 MDT

In my experience, NCL reads an attribute without reading the whole
file variable, so normally this strategy should be much faster.

However, there is a quirk which may be related. When you first open
an input file with addfile, NCL invisibly reads all *coordinate
variables* into memory. This can take surprisingly long if the
coordinate values are scattered throughout the file, in particular for
large Netcdf-3 format files with a coordinate variable on the
unlimited dimension. I have not looked into this for Netcdf-4/HDF
format, but this may be mitigated by chunking strategy.

So, any performance testing of this nature should segregate the
latency for addfile from the desired timing to be measured.

--Dave

On Wed, Jul 18, 2012 at 2:02 PM, David B. Reusch <dreusch@ees.nmt.edu> wrote:
> Thanks Dennis but that's the answer to a different question from what I
> had in mind. I was just trying to confirm whether NCL is reading in the
> whole variable when performing an operation to get an attribute, like
> the example you provided. My understanding of the netCDF library is
> that it shouldn't have to. But I'm either misunderstanding or NCL is
> reading it all since my few tests have taken about the same time to read
> the var or read its attribute. I was hoping an up-front investment to
> read the var/save the stats as attributes would pay off later when I
> just read the attributes.
>
> If I haven't already spent too much time (mine and yours) on this, I
> suspect I'll be there soon...
>
> Thanks,
> Dave
>
> Dennis Shea said the following on 7/18/12 1:54 PM:
>> f = addfile("foo.nc", "r")
>>
>> Q_MinMax = f->Q@min_max
>>
>>
>> On 7/18/12 1:36 PM, David B. Reusch wrote:
>>> Hello all,
>>>
>>> I have this idea that by adding min/max values for variables as
>>> attributes in the netCDF file I might be able to save time later when
>>> trying to create consistent colorbars/contouring, i.e., my script just
>>> reads the attribute instead of crunching through all the data. Some
>>> experience has suggested I won't actually be saving anything since it
>>> seems to be taking just as long to read the attributes as it does to
>>> read the whole var (though it will still be nice to have the stats
>>> easily viewable as attributes...). Hence my actual question: does NCL
>>> have to read the whole var to get its attributes? I wouldn't have
>>> thought the netCDF data model would require this but I'm not an expert.
>>> And it's also possible something else is still slowing it down, I
>>> haven't had a chance to really dig into this much.
>>>
>>> Thanks,
>>> Dave
>>>
>>
>>
>
> --
> Associate Research Professor of Climatology
> Dept of Earth and Environmental Science
> MSEC 304; 801 Leroy Place
> New Mexico Tech
> Socorro, NM 87801
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Wed Jul 18 16:10:31 2012

This archive was generated by hypermail 2.1.8 : Tue Jul 24 2012 - 09:48:26 MDT