Re: large files

Hi Mike,
The most significant difference between the two files that I can see is
that in cru_ts_2_10.1901-2002.pre.nc the time dimension is 'unlimited',
whereas in cru_ts_2_10.1901-2002.pre.climo.repeat.jja.nc it is a regular
fixed size dimension. There are significant differences in the storage
and
retrieval strategies that the NetCDF library uses depending on whether
a dimension
is 'unlimited'. I am somewhat surprised to see that the file with the
unlimited
dimension has better performance on the Mac. I might have expected it
to be the
other way around, and this could be worth investigating. But in the
meantime, in
answer to your last question, I think if you want to get the same
performance
for each file, you should make them both the same with respect to the
'unlimited'
dimension.

How much RAM do you have on your Mac? That, of course, is a signifcant
difference
between your Mac and tempest.
  -dave
On Apr 23, 2007, at 10:46 AM, Michael Notaro wrote:

> I have 2 netcdf files, both 1.3 GB. The NCL script, openfile.ncl, and
> the two
> nc files are on tempest at /ptmp/notaro/testing . I also have copies
> on my mac.
>
> When I open file cru_ts_2_10.1901-2002.pre.nc on my mac and retrieve
> the variable
> "pre", it takes < 1 minute. However, if I open the file
> cru_ts_2_10.1901-2002.pre.climo.repeat.jja.nc
> and retrieve "pre" on my mac, it takes an eternity. The 2 files are
> almost identical in size.
>
> Strangely enough, on tempest, both are fast.
>
> Any idea what is wrong? Something to do with attributes/FillValue/etc?
> How can I make the slow file open faster?
>
> Thanks,
> Mike
> _______________________________________________
> ncl-talk mailing list
> ncl-talk_at_ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
ncl-talk_at_ucar.edu
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Mon Apr 23 2007 - 12:28:19 MDT