Re: addfile delay

Hi Dave,

Yes, in fact NCL does read all the coordinate variables and cache their
values in memory and at present there is no way to turn this behavior
off.

Most likely, as Dennis suggests, you have a large file that has an
unlimited
dimension with a coordinate variable (most usually a time coordinate).
In this case the NetCDF library needs to read every 'record' to get the
coordinate value associated with that element of the unlimited
dimension.

It used to be that NCL would re-read the coordinate variable with every
access. At least with the coordinate variable caching that it now does,
it only has to read the values once.

The only solution (that I can think of immediately), which may not be
practical,
is to rewrite the file without an unlimited dimension, or perhaps
disassociate
the unlimited dimension from its coordinate value (by using different
names
for the dimension and the coordinate value), and then provide an
auxiliary
variable (perhaps in another file) that would contain the coordinate
values.

Again, if you like we can take a look at your file.
  -dave

On Jan 16, 2007, at 10:49 AM, Dennis Shea wrote:

> fyi
> ------------- Begin Forwarded Message -------------
>
> Date: Tue, 16 Jan 2007 10:38:04 -0700
> From: Dave Allured <dave.allured_at_noaa.gov>
> Subject: Re: addfile delay
> To: Dennis Shea <shea_at_ucar.edu>
> MIME-version: 1.0
> Content-transfer-encoding: 7BIT
> User-Agent: Thunderbird 1.5.0.8 (Macintosh/20061025)
> X-Virus-Scanned: amavisd-new at ucar.edu
>
> Dennis,
>
> Yes that is the form of addfile that I am using. I should mention
> that the file is local, not Opendap.
>
> Yes, I believe we have large file support. My file is a Netcdf
> "classic" format with 32-bit offsets. The record variables extend
> past 2 Gb, but Netcdf documentation says that this is legitimate.
>
> Before expending time on file upload and analysis, I would like to
> see if the reply from Dave Brown is forthcoming. Reading the time
> coordinate variable would explain everything.
>
> Thank you for your reply.
>
> --Dave
>
> Dennis Shea wrote:
>> offline
>>
>> I assume you mean:
>>
>> f = addfile ("foo.nc", "r")
>>
>>> 1. Does the addfile function by itself read coordinate variables
>>> into
> memory?
>>
>> No. I believe ... For netCDf it just opens the file.
>>
>>> If not, why does addfile take a long time to execute on a large
>>> Netcdf file
>>> (e.g. greater than 2 Gb)?
>>
>> I have no idea. I assume you have Large File Support.
>>
>>> 2. Is there a simple way to turn off this behavior? No fair
>>> patching NCL.
>>> My NCL version is 4.2.0.a034, running on MacOS/Darwin 8.8.0.
>>
>> There is no behavior to turn off.
>>
>> I'm sure Dave Brown will answer.
>>
>> If you want, you can ftp the file to
>>
>> ftp ftp.cgd.ucar.edu
>> anonymous
>> email
>> cd incoming
>> put .....
>> quit
>>
>> Let us know if you do this.
>>
>> D
>
> ------------- End Forwarded Message -------------
>
>

_______________________________________________
ncl-talk mailing list
ncl-talk_at_ucar.edu
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Tue Jan 16 2007 - 11:30:16 MST