Re: reading in large netcdf files

All tools (NCL, IDL, Matlab, ferret, GrADS, ...) use the same
Unidata software. As noted by Russ Rew of Unidata, the software
is written for reliability and robustness, not necessarily speed.

Still, 30-45 min seems a *really* slow. It looks like you are
getting *one* value for prcp and *one* value for z850 from
a file of 25GB in size. If you are in a loop do this for the 15 years
of data [15*365] reads, I can see why it is slow. That is not what
the Unidata software was designed to do!

   yrStrt = ....
   yrLast = ....
   infile = addfile("...", "r")
   time = infile->time ; "??? since ....??? "
   ymd = cd_calendar(time, -2) ; yyyymmdd
   year = ymd/10000
   iyear = ind(year.ge.yrStrt .and. year.le.yrLast)
          ; get all time steps at one time
   prcp = infile->pr(iyear,{lat:lat},{lon:lon})
   z850 = infile->hgt(iyear,{lev},{lat:lat},{lon:lon})

   printVarSummary(prcp)
   printVarSummary(z850)

==
Another approach is to use the netCDF operators (ncrcat, ncks)
to create a file with only those two variables on it.

   %> ncrcat -v pr,hgt -d time,iStrt,iLast filename.nc foo.nc

Good Luck

On 10/19/11 11:12 PM, Kerrie Geil wrote:
> Hello,
>
> What's the fastest way to read in a variable from a large netcdf file?
> Say I have a 25GB file (or larger) containing over 100 years worth of 3D
> and 4D daily atmospheric variables and I only want to read in 2 of these
> variables over 15 years or so. What I do now is subset everything I can
> like the following:
>
> infile=addfile("filename","r")
> prcp=infile->pr(index:index,{lat:lat},{lon:lon})
> z850=infile->hgt(index:index,{lev},{lat:lat},{lon:lon})
>
> For large files though, this step alone can take up to 30-45min
> everytime I run the script. Is there any procedure similar to
> setfileoption that I can use to speed up the reading process? Or maybe
> there's some other work around? I know this is not a problem with some
> other programs, but I happen not to like these programs as much as NCL.
>
> Thanks,
>
> Kerrie Geil
> Master's Student
> Department of Atmospheric Sciences
> University of Arizona
> PAS Building Rm 526
> 1118 E 4th Street
> Tucson, AZ 85721
>
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Thu Oct 20 06:59:13 2011