Re: reading all the variables from a netCDF when the variable names are unknown (or too many)

Hi Saji,
    Thanks for the suggestion - that is an innovative idea. I was hoping
to modify this idea a bit and write code to generate just the lines of
code I would need to read in each variable, write these out as a module,
and then load that module within my program in the spot I need it - but
unfortunately "load" only works if the module is loaded outside of the
block that is planning to use it, so I get an error when I try to load it:
"input in flex scanner failed"

So for now, I'm just going to use NCL to generate the 100+ lines to
reach each variable and cut and paste it into my program (and repeat
this if the variables in my netCDF files change).

Hopefully new versions of NCL can include some of these nifty features,
like an "eval" function, a way to create variables without assignment,
and a way to reference a variable without knowing it's name before runtime.

Jonathan

Saji N. Hameed wrote:
> Hi Jonathan,
>
> As a follow up of my previous mail related to "eval", why don't you
> run ncl on your code twice? The first pass will create NCL code that
> will be run in the second pass.
>
> Here is my crude semi-pseudo-code
>
> Start of NCL file ----------
>
> _NCLCode="" ; declared at the outset. global variable??
>
> function cr()
> begin
> return(inttochar(10))
> end
>
> procedure add_msg(msg)
> begin
> _NCLCode=_NCLCode+cr+msg
> end
>
>
> begin
> do ivar = 0,nvar-1
> msg =varnames(ivar)+" = fin->$"+varnames(ivar)+"$"
> add_msg(msg)
> end do
>
> asciiwrite("runit.ncl",_NCLCode)
> system("ncl runit.ncl")
> end
>
>
>
> * Jonathan Vigh <vigh_at_atmos.colostate.edu> [2009-02-24 21:05:59 +0000]:
>
>
>> Hi David,
>> I'll likely be doing some conditional processing, where some
>> variables affect others, so I'd like to have them all in memory at once.
>> For this specific application, all the variables are quite small - the
>> entire netCDF file is <50 KB - so it won't be a problem to hold them all
>> in memory. I'm planning to read in about 200 of these files and
>> aggregate some of the 177 variables into several multidimensional arrays
>> for further data analysis. The variables are a hodgepodge of arrays of
>> strings, logicals, integers, floats, etc. It's a mess. I've never had to
>> work with such a disparate collection of data before. The netCDF files
>> I'm reading are actually an intermediate (but necessary) step - the data
>> were originally read from several very nonstandard ascii file formats
>> and I wrote them to the netCDF files. I wrote out the 177 variables
>> using a general method, so that's why I was hoping to read in the 177
>> variables in a general way. The trouble with creating the variable
>> without knowing it's name seems to be the main hitch at the moment.
>> Accessing it without knowing it's name is the other hitch.
>>
>> So basically, I think need a "create_new_var" procedure (not function)
>> which would act like:
>> create_new_var( varname, vartype, vardimsize )
>>
>> One the variable is created in memory, I'd need a way to reference it in
>> order to assign data to it from the file.
>>
>> So my code to read in everything would be something like:
>> varnames = getfilevarnames(fin) ; Returns an array of file
>> variable names in the file
>> vartypes = getfilevartypes(fin,varnames) ; Returns the type of
>> each variable name listed. A missing value is returned for any variable
>> name that doesn't exist in the file.
>>
>> nvars = dimsizes(varnames)
>> do ivar = 0, nvars-1
>> create_new_var( varnames(ivar), vartypes(ivar),
>> getfilevardimsizes(fin,varnames(ivar)) )
>> #varnames(ivar)# = fin->$varnames(ivar)$ ; where # # is used
>> to indicate the way to reference the variable in memory using the string
>> value from my array varnames that was queried from the file
>> end do
>>
>> If there was a way to reference the variable from memory, one could do
>> all the processing on an arbitrary variable without ever knowing it's
>> name. This would be a nice feature to have.
>>
>> Jonathan
>>
>>
>>
>> Dave Allured wrote:
>>
>>> Jonathan,
>>>
>>> Do you need to have all the variables in memory at the same time, or
>>> can you just process them one at a time from the input file to the
>>> output file(s)? The second has a much easier answer than the first.
>>> A little more information about the nature of your application is
>>> needed.
>>>
>>> Dave Allured
>>> CU/CIRES Climate Diagnostics Center (CDC)
>>> http://cires.colorado.edu/science/centers/cdc/
>>> NOAA/ESRL/PSD, Climate Analysis Branch (CAB)
>>> http://www.cdc.noaa.gov/psd1/
>>>
>>> Jonathan Vigh wrote:
>>>
>>>
>>>> Greetings NCL'ers,
>>>> I'd like to read in all the variables from a netCDF file in a
>>>> general way because there are 177 of them. I understand how to query the
>>>> file and get a list of variable names, and I even understand how to
>>>> reference the variable using:
>>>> fin->$varnames$
>>>>
>>>> What I am confused about is what to assign this to. I'd like to avoid
>>>> having to type out 177 lines to read in each variable if possible. But
>>>> if I didn't even know the variable names in advance, I'd still like to
>>>> know how to do this.
>>>>
>>>> Maybe I've missed something really basic, but I couldn't seem to find
>>>> anything from the documentation. If anyone has an idea, I'd be most
>>>> grateful. It would be nice if NCL could simply make all the variables in
>>>> a netCDF file available without having to go through this step, so maybe
>>>> this could be the prototype for a new function that does this.
>>>>
>>>> Thanks,
>>>> Jonathan
>>>>
>>>>
>>>> begin
>>>> processed_filename = systemfunc("cd " + processed_data_directory + ";
>>>> ls " + stormid + "*.nc")
>>>> fin = addfile(processed_data_directory + processed_filename,"r") ;
>>>> open output netCDF file
>>>>
>>>> ; query the file to see what variables it contains and what the sizes of
>>>> the file dimensions are
>>>> varnames = getfilevarnames(fin) ; Returns an array of file
>>>> variable names in the file
>>>> filedimsizes = getfiledimsizes(fin) ; Returns a list of the
>>>> sizes of all the dimensions in the file.
>>>>
>>>> ; query all the named variables in the file
>>>> vartypes = getfilevartypes(fin,varnames) ; Returns the type of
>>>> each variable name listed. A missing value is returned for any variable
>>>> name that doesn't exist in the file.
>>>>
>>>>
>>>> ; loop through all the variables, query them, and read them in
>>>> nvars = dimsizes(varnames)
>>>> do ivar = 0, nvars-1
>>>> print("Reading variable ivar = "+ivar+" "+vartypes(ivar)+"
>>>> "+varnames(ivar))
>>>> ???? = fin->$varnames(ivar)$ ; I understand how to
>>>> read in a variable from a file if I don't know the variable name in
>>>> advance, but how can I assign it to a variable without typing the
>>>> variable's name?
>>>> end do
>>>>
>>>> end
>>>> _______________________________________________
>>>> ncl-talk mailing list
>>>> List instructions, subscriber options, unsubscribe:
>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>
>>>>
>>> _______________________________________________
>>> ncl-talk mailing list
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>>
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>
>

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Thu Feb 26 2009 - 13:15:18 MST