Re: most efficient way to manipulate a large data set with NCL

[1]

Not sure of the details of your files. Do they already
have a '.grb' extension? If not, you can manually add it to the script
as is done below. Otherwise, remove the references to ".grb" below

[2]

Option 1: Something like the following. The 'setfileoption'
           forces a time dimension if there is none.

   diri = ".../"
   fRoot = "FOO..." ; grib file root
   setfileoption("grb","SingleElementDimensions","Initial_time")

        fili = systemfunc("cd "+diri+" ; ls "+fRoot+"*")
        print(fili)

        f = addfiles( diri+fili+".grb", "r")

        x = f[:]->????? ; use default 'cat' mode
        printVarSummary(x)
-----------

Option 2: don't know if what your student says is true.

-----------
Option 3: Create a netCDF file of each grib.
Maybe you need v6.1.0-beta for this? Not sure.

http://www.ncl.ucar.edu/Document/Tools/ncl_convert2nc.shtml

[a]
ncl_convert2nc FOO* -e grb -itime -u initial_time0_hours -U time

[b]
ncrcat FOO*nc cat.FOO.nc

Then work on the one file.

On 9/12/12 1:42 PM, Serra, Yolande L - (serra) wrote:
> Hello -
>
> I'm not a "big" NCL user and am in need of some advice on reading a
> large data set. What is the most efficient way to read in a 4-D array,
> manipulate the data in the time and level dimensions and re-write to a
> new file? The catch is that the variables are stored as 3-D, with each
> time step being stored as a separate grib file at the moment.
>
> So far as I can tell my options are:
>
> 1) use *addfiles* and read each individual time into the code using
> "join" to create the new time dimension. The variable would then be
> 4-D and could be manipulated inside NCL. The problem is this seems to
> be extremely slow just reading the files and loading the variable.
>
> 2) use a loop over time and load in files one at a time using *addfile*
> - one student of mine claimed this is faster than using *addfiles*. Is
> this true?
>
> 3) use *ncl_convert2nc* to convert the grib files to netcdf, concatenate
> to one file with a time dimension and read that file into NCL. Is this
> faster than using *addfiles*?
>
> Why does it take NCL so much longer to read a grib or netcdf file than
> Matlab?
>
> The final 4-D variable is 489 Times x 60 Levels x 85 Lats x 512 Lons.
> The manipulations I need to perform are to fit to pressure levels from
> model levels and then low-pass filter in time. This part of the code
> seems to be working fine, the slow part is loading the data.
>
> Thanks for any help, Yolande
>
> Yolande Serra
> serra@email.arizona.edu <mailto:serra@email.arizona.edu>
>
>
>
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Wed Sep 12 14:52:19 2012