Re: reading all the variables from a netCDF when the variable names are unknown (or too many)

Hi Jonathan,

>#varnames(ivar)# = fin->$varnames(ivar)$ ; where # # is used to
>indicate the way to reference the variable in memory using the string
>value from my array varnames that was queried from the file

Unfortunately, it isn't currently possible, but it has been categorized
as a high priority item by the NCL development team. Best case scenario
would be that it will be implemented by this fall.

I agree, this would be a great enhancement to NCL..
Adam

Jonathan Vigh wrote:
> Hi David,
> I'll likely be doing some conditional processing, where some
> variables affect others, so I'd like to have them all in memory at once.
> For this specific application, all the variables are quite small - the
> entire netCDF file is <50 KB - so it won't be a problem to hold them all
> in memory. I'm planning to read in about 200 of these files and
> aggregate some of the 177 variables into several multidimensional arrays
> for further data analysis. The variables are a hodgepodge of arrays of
> strings, logicals, integers, floats, etc. It's a mess. I've never had to
> work with such a disparate collection of data before. The netCDF files
> I'm reading are actually an intermediate (but necessary) step - the data
> were originally read from several very nonstandard ascii file formats
> and I wrote them to the netCDF files. I wrote out the 177 variables
> using a general method, so that's why I was hoping to read in the 177
> variables in a general way. The trouble with creating the variable
> without knowing it's name seems to be the main hitch at the moment.
> Accessing it without knowing it's name is the other hitch.
>
> So basically, I think need a "create_new_var" procedure (not function)
> which would act like:
> create_new_var( varname, vartype, vardimsize )
>
> One the variable is created in memory, I'd need a way to reference it in
> order to assign data to it from the file.
>
> So my code to read in everything would be something like:
> varnames = getfilevarnames(fin) ; Returns an array of file
> variable names in the file
> vartypes = getfilevartypes(fin,varnames) ; Returns the type of
> each variable name listed. A missing value is returned for any variable
> name that doesn't exist in the file.
>
> nvars = dimsizes(varnames)
> do ivar = 0, nvars-1
> create_new_var( varnames(ivar), vartypes(ivar),
> getfilevardimsizes(fin,varnames(ivar)) )
> #varnames(ivar)# = fin->$varnames(ivar)$ ; where # # is used
> to indicate the way to reference the variable in memory using the string
> value from my array varnames that was queried from the file
> end do
>
> If there was a way to reference the variable from memory, one could do
> all the processing on an arbitrary variable without ever knowing it's
> name. This would be a nice feature to have.
>
> Jonathan
>
>
>
> Dave Allured wrote:
>> Jonathan,
>>
>> Do you need to have all the variables in memory at the same time, or
>> can you just process them one at a time from the input file to the
>> output file(s)? The second has a much easier answer than the first.
>> A little more information about the nature of your application is
>> needed.
>>
>> Dave Allured
>> CU/CIRES Climate Diagnostics Center (CDC)
>> http://cires.colorado.edu/science/centers/cdc/
>> NOAA/ESRL/PSD, Climate Analysis Branch (CAB)
>> http://www.cdc.noaa.gov/psd1/
>>
>> Jonathan Vigh wrote:
>>
>>> Greetings NCL'ers,
>>> I'd like to read in all the variables from a netCDF file in a
>>> general way because there are 177 of them. I understand how to query the
>>> file and get a list of variable names, and I even understand how to
>>> reference the variable using:
>>> fin->$varnames$
>>>
>>> What I am confused about is what to assign this to. I'd like to avoid
>>> having to type out 177 lines to read in each variable if possible. But
>>> if I didn't even know the variable names in advance, I'd still like to
>>> know how to do this.
>>>
>>> Maybe I've missed something really basic, but I couldn't seem to find
>>> anything from the documentation. If anyone has an idea, I'd be most
>>> grateful. It would be nice if NCL could simply make all the variables in
>>> a netCDF file available without having to go through this step, so maybe
>>> this could be the prototype for a new function that does this.
>>>
>>> Thanks,
>>> Jonathan
>>>
>>>
>>> begin
>>> processed_filename = systemfunc("cd " + processed_data_directory + ";
>>> ls " + stormid + "*.nc")
>>> fin = addfile(processed_data_directory + processed_filename,"r") ;
>>> open output netCDF file
>>>
>>> ; query the file to see what variables it contains and what the sizes of
>>> the file dimensions are
>>> varnames = getfilevarnames(fin) ; Returns an array of file
>>> variable names in the file
>>> filedimsizes = getfiledimsizes(fin) ; Returns a list of the
>>> sizes of all the dimensions in the file.
>>>
>>> ; query all the named variables in the file
>>> vartypes = getfilevartypes(fin,varnames) ; Returns the type of
>>> each variable name listed. A missing value is returned for any variable
>>> name that doesn't exist in the file.
>>>
>>>
>>> ; loop through all the variables, query them, and read them in
>>> nvars = dimsizes(varnames)
>>> do ivar = 0, nvars-1
>>> print("Reading variable ivar = "+ivar+" "+vartypes(ivar)+"
>>> "+varnames(ivar))
>>> ???? = fin->$varnames(ivar)$ ; I understand how to
>>> read in a variable from a file if I don't know the variable name in
>>> advance, but how can I assign it to a variable without typing the
>>> variable's name?
>>> end do
>>>
>>> end
>>> _______________________________________________
>>> ncl-talk mailing list
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

-- 
--------------------------------------------------------------
Adam Phillips			             asphilli_at_ucar.edu
National Center for Atmospheric Research   tel: (303) 497-1726
ESSL/CGD/CAS                               fax: (303) 497-1333
P.O. Box 3000				
Boulder, CO 80307-3000	  http://www.cgd.ucar.edu/cas/asphilli
_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk