Re: addfile delay

Thanks Dave and Dennis for your replies. Now I understand what is
going on with addfile. It is not a big deal, it just slows down my
debug cycles. "I thought I might be doing something wrong." ;-)

Please consider adding a footnote on the addfile page about caching
the coordinate variables.

--Dave

David Ian Brown wrote:
> Hi Dave,
>
> Yes, in fact NCL does read all the coordinate variables and cache their
> values in memory and at present there is no way to turn this behavior off.
>
> Most likely, as Dennis suggests, you have a large file that has an
> unlimited
> dimension with a coordinate variable (most usually a time coordinate).
> In this case the NetCDF library needs to read every 'record' to get the
> coordinate value associated with that element of the unlimited dimension.
>
> It used to be that NCL would re-read the coordinate variable with every
> access. At least with the coordinate variable caching that it now does,
> it only has to read the values once.
>
> The only solution (that I can think of immediately), which may not be
> practical,
> is to rewrite the file without an unlimited dimension, or perhaps
> disassociate
> the unlimited dimension from its coordinate value (by using different names
> for the dimension and the coordinate value), and then provide an auxiliary
> variable (perhaps in another file) that would contain the coordinate
> values.
>
> Again, if you like we can take a look at your file.
> -dave
>
>
> On Jan 16, 2007, at 10:49 AM, Dennis Shea wrote:
>
>> fyi
>> ------------- Begin Forwarded Message -------------
>>
>> Date: Tue, 16 Jan 2007 10:38:04 -0700
>> From: Dave Allured <dave.allured_at_noaa.gov>
>> Subject: Re: addfile delay
>> To: Dennis Shea <shea_at_ucar.edu>
>> MIME-version: 1.0
>> Content-transfer-encoding: 7BIT
>> User-Agent: Thunderbird 1.5.0.8 (Macintosh/20061025)
>> X-Virus-Scanned: amavisd-new at ucar.edu
>>
>> Dennis,
>>
>> Yes that is the form of addfile that I am using. I should mention
>> that the file is local, not Opendap.
>>
>> Yes, I believe we have large file support. My file is a Netcdf
>> "classic" format with 32-bit offsets. The record variables extend
>> past 2 Gb, but Netcdf documentation says that this is legitimate.
>>
>> Before expending time on file upload and analysis, I would like to
>> see if the reply from Dave Brown is forthcoming. Reading the time
>> coordinate variable would explain everything.
>>
>> Thank you for your reply.
>>
>> --Dave
>>
>> Dennis Shea wrote:
>>> offline
>>>
>>> I assume you mean:
>>>
>>> f = addfile ("foo.nc", "r")
>>>
>>>> 1. Does the addfile function by itself read coordinate variables into
>> memory?
>>>
>>> No. I believe ... For netCDf it just opens the file.
>>>
>>>> If not, why does addfile take a long time to execute on a large
>>>> Netcdf file
>>>> (e.g. greater than 2 Gb)?
>>>
>>> I have no idea. I assume you have Large File Support.
>>>
>>>> 2. Is there a simple way to turn off this behavior? No fair
>>>> patching NCL.
>>>> My NCL version is 4.2.0.a034, running on MacOS/Darwin 8.8.0.
>>>
>>> There is no behavior to turn off.
>>>
>>> I'm sure Dave Brown will answer.
>>>
>>> If you want, you can ftp the file to
>>>
>>> ftp ftp.cgd.ucar.edu
>>> anonymous
>>> email
>>> cd incoming
>>> put .....
>>> quit
>>>
>>> Let us know if you do this.
>>>
>>> D
>>
>> ------------- End Forwarded Message -------------
>>
>>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk_at_ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
ncl-talk_at_ucar.edu
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Tue Jan 16 2007 - 12:19:48 MST