Re: large files

I tried creating the file without FillValue and using time unlimited
and now
it is very fast.
Thanks.

Mike

On Apr 23, 2007, at 1:28 PM, David Ian Brown wrote:

>
> Hi Mike,
> The most significant difference between the two files that I can
> see is
> that in cru_ts_2_10.1901-2002.pre.nc the time dimension is
> 'unlimited',
> whereas in cru_ts_2_10.1901-2002.pre.climo.repeat.jja.nc it is a
> regular
> fixed size dimension. There are significant differences in the
> storage and
> retrieval strategies that the NetCDF library uses depending on
> whether a dimension
> is 'unlimited'. I am somewhat surprised to see that the file with
> the unlimited
> dimension has better performance on the Mac. I might have expected
> it to be the
> other way around, and this could be worth investigating. But in the
> meantime, in
> answer to your last question, I think if you want to get the same
> performance
> for each file, you should make them both the same with respect to
> the 'unlimited'
> dimension.
>
> How much RAM do you have on your Mac? That, of course, is a
> signifcant difference
> between your Mac and tempest.
> -dave
> On Apr 23, 2007, at 10:46 AM, Michael Notaro wrote:
>
>> I have 2 netcdf files, both 1.3 GB. The NCL script, openfile.ncl,
>> and the two
>> nc files are on tempest at /ptmp/notaro/testing . I also have
>> copies on my mac.
>>
>> When I open file cru_ts_2_10.1901-2002.pre.nc on my mac and
>> retrieve the variable
>> "pre", it takes < 1 minute. However, if I open the file
>> cru_ts_2_10.1901-2002.pre.climo.repeat.jja.nc
>> and retrieve "pre" on my mac, it takes an eternity. The 2 files
>> are almost identical in size.
>>
>> Strangely enough, on tempest, both are fast.
>>
>> Any idea what is wrong? Something to do with attributes/FillValue/
>> etc?
>> How can I make the slow file open faster?
>>
>> Thanks,
>> Mike
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk_at_ucar.edu
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>

_______________________________________________
ncl-talk mailing list
ncl-talk_at_ucar.edu
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Mon Apr 23 2007 - 14:58:57 MDT