Re: reading all the variables from a netCDF when the variable names are unknown (or too many)

Hi David,
   I'll likely be doing some conditional processing, where some
variables affect others, so I'd like to have them all in memory at once.
For this specific application, all the variables are quite small - the
entire netCDF file is <50 KB - so it won't be a problem to hold them all
in memory. I'm planning to read in about 200 of these files and
aggregate some of the 177 variables into several multidimensional arrays
for further data analysis. The variables are a hodgepodge of arrays of
strings, logicals, integers, floats, etc. It's a mess. I've never had to
work with such a disparate collection of data before. The netCDF files
I'm reading are actually an intermediate (but necessary) step - the data
were originally read from several very nonstandard ascii file formats
and I wrote them to the netCDF files. I wrote out the 177 variables
using a general method, so that's why I was hoping to read in the 177
variables in a general way. The trouble with creating the variable
without knowing it's name seems to be the main hitch at the moment.
Accessing it without knowing it's name is the other hitch.

So basically, I think need a "create_new_var" procedure (not function)
which would act like:
   create_new_var( varname, vartype, vardimsize )

One the variable is created in memory, I'd need a way to reference it in
order to assign data to it from the file.

So my code to read in everything would be something like:
  varnames = getfilevarnames(fin) ; Returns an array of file
variable names in the file
  vartypes = getfilevartypes(fin,varnames) ; Returns the type of
each variable name listed. A missing value is returned for any variable
name that doesn't exist in the file.

  nvars = dimsizes(varnames)
   do ivar = 0, nvars-1
      create_new_var( varnames(ivar), vartypes(ivar),
getfilevardimsizes(fin,varnames(ivar)) )
      #varnames(ivar)# = fin->$varnames(ivar)$ ; where # # is used
to indicate the way to reference the variable in memory using the string
value from my array varnames that was queried from the file
   end do

If there was a way to reference the variable from memory, one could do
all the processing on an arbitrary variable without ever knowing it's
name. This would be a nice feature to have.

Jonathan

Dave Allured wrote:
> Jonathan,
>
> Do you need to have all the variables in memory at the same time, or
> can you just process them one at a time from the input file to the
> output file(s)? The second has a much easier answer than the first.
> A little more information about the nature of your application is
> needed.
>
> Dave Allured
> CU/CIRES Climate Diagnostics Center (CDC)
> http://cires.colorado.edu/science/centers/cdc/
> NOAA/ESRL/PSD, Climate Analysis Branch (CAB)
> http://www.cdc.noaa.gov/psd1/
>
> Jonathan Vigh wrote:
>
>> Greetings NCL'ers,
>> I'd like to read in all the variables from a netCDF file in a
>> general way because there are 177 of them. I understand how to query the
>> file and get a list of variable names, and I even understand how to
>> reference the variable using:
>> fin->$varnames$
>>
>> What I am confused about is what to assign this to. I'd like to avoid
>> having to type out 177 lines to read in each variable if possible. But
>> if I didn't even know the variable names in advance, I'd still like to
>> know how to do this.
>>
>> Maybe I've missed something really basic, but I couldn't seem to find
>> anything from the documentation. If anyone has an idea, I'd be most
>> grateful. It would be nice if NCL could simply make all the variables in
>> a netCDF file available without having to go through this step, so maybe
>> this could be the prototype for a new function that does this.
>>
>> Thanks,
>> Jonathan
>>
>>
>> begin
>> processed_filename = systemfunc("cd " + processed_data_directory + ";
>> ls " + stormid + "*.nc")
>> fin = addfile(processed_data_directory + processed_filename,"r") ;
>> open output netCDF file
>>
>> ; query the file to see what variables it contains and what the sizes of
>> the file dimensions are
>> varnames = getfilevarnames(fin) ; Returns an array of file
>> variable names in the file
>> filedimsizes = getfiledimsizes(fin) ; Returns a list of the
>> sizes of all the dimensions in the file.
>>
>> ; query all the named variables in the file
>> vartypes = getfilevartypes(fin,varnames) ; Returns the type of
>> each variable name listed. A missing value is returned for any variable
>> name that doesn't exist in the file.
>>
>>
>> ; loop through all the variables, query them, and read them in
>> nvars = dimsizes(varnames)
>> do ivar = 0, nvars-1
>> print("Reading variable ivar = "+ivar+" "+vartypes(ivar)+"
>> "+varnames(ivar))
>> ???? = fin->$varnames(ivar)$ ; I understand how to
>> read in a variable from a file if I don't know the variable name in
>> advance, but how can I assign it to a variable without typing the
>> variable's name?
>> end do
>>
>> end
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Tue Feb 24 2009 - 14:05:59 MST