Re: reading all variables from a netCDF file - solved

Hi all,
   After thinking some more, I realized a way to incorporate Saji's
suggestion as a module that runs at the beginning of the program. This
module opens the file and queries it to obtain the list of variables.
Using that variable list, it dynamically generates another module which
contains the lines to read them from the file. This module should be
loaded immediately after the first one. Both modules are called from the
main calling program outside of the main statement block, so all the
variables read in from the file will be global and available to all
functions and procedures throughout the program.

Since all that is required is to load the two modules at the top of your
program, I think this is a pretty unobtrusive way to read in the
contents of a netCDF file with many or unknown variables.

I've attached the module_read_netcdf.ncl code, the module that was
generated dynamically (which will be overwritten at run-time), an
example 'main.ncl', and a simple netCDF file for those who are curious
to try this out.

Thanks for the helpful suggestion!
 
Jonathan

Jonathan Vigh wrote:
> Hi Saji,
> Thanks for the suggestion - that is an innovative idea. I was hoping
> to modify this idea a bit and write code to generate just the lines of
> code I would need to read in each variable, write these out as a module,
> and then load that module within my program in the spot I need it - but
> unfortunately "load" only works if the module is loaded outside of the
> block that is planning to use it, so I get an error when I try to load it:
> "input in flex scanner failed"
>
> So for now, I'm just going to use NCL to generate the 100+ lines to
> reach each variable and cut and paste it into my program (and repeat
> this if the variables in my netCDF files change).
>
> Hopefully new versions of NCL can include some of these nifty features,
> like an "eval" function, a way to create variables without assignment,
> and a way to reference a variable without knowing it's name before runtime.
>
> Jonathan
>
>
>
>
>
> Saji N. Hameed wrote:
>
>> Hi Jonathan,
>>
>> As a follow up of my previous mail related to "eval", why don't you
>> run ncl on your code twice? The first pass will create NCL code that
>> will be run in the second pass.
>>
>> Here is my crude semi-pseudo-code
>>
>> Start of NCL file ----------
>>
>> _NCLCode="" ; declared at the outset. global variable??
>>
>> function cr()
>> begin
>> return(inttochar(10))
>> end
>>
>> procedure add_msg(msg)
>> begin
>> _NCLCode=_NCLCode+cr+msg
>> end
>>
>>
>> begin
>> do ivar = 0,nvar-1
>> msg =varnames(ivar)+" = fin->$"+varnames(ivar)+"$"
>> add_msg(msg)
>> end do
>>
>> asciiwrite("runit.ncl",_NCLCode)
>> system("ncl runit.ncl")
>> end
>>
>>
>>
>> * Jonathan Vigh <vigh_at_atmos.colostate.edu> [2009-02-24 21:05:59 +0000]:
>>
>>
>>
>>> Hi David,
>>> I'll likely be doing some conditional processing, where some
>>> variables affect others, so I'd like to have them all in memory at once.
>>> For this specific application, all the variables are quite small - the
>>> entire netCDF file is <50 KB - so it won't be a problem to hold them all
>>> in memory. I'm planning to read in about 200 of these files and
>>> aggregate some of the 177 variables into several multidimensional arrays
>>> for further data analysis. The variables are a hodgepodge of arrays of
>>> strings, logicals, integers, floats, etc. It's a mess. I've never had to
>>> work with such a disparate collection of data before. The netCDF files
>>> I'm reading are actually an intermediate (but necessary) step - the data
>>> were originally read from several very nonstandard ascii file formats
>>> and I wrote them to the netCDF files. I wrote out the 177 variables
>>> using a general method, so that's why I was hoping to read in the 177
>>> variables in a general way. The trouble with creating the variable
>>> without knowing it's name seems to be the main hitch at the moment.
>>> Accessing it without knowing it's name is the other hitch.
>>>
>>> So basically, I think need a "create_new_var" procedure (not function)
>>> which would act like:
>>> create_new_var( varname, vartype, vardimsize )
>>>
>>> One the variable is created in memory, I'd need a way to reference it in
>>> order to assign data to it from the file.
>>>
>>> So my code to read in everything would be something like:
>>> varnames = getfilevarnames(fin) ; Returns an array of file
>>> variable names in the file
>>> vartypes = getfilevartypes(fin,varnames) ; Returns the type of
>>> each variable name listed. A missing value is returned for any variable
>>> name that doesn't exist in the file.
>>>
>>> nvars = dimsizes(varnames)
>>> do ivar = 0, nvars-1
>>> create_new_var( varnames(ivar), vartypes(ivar),
>>> getfilevardimsizes(fin,varnames(ivar)) )
>>> #varnames(ivar)# = fin->$varnames(ivar)$ ; where # # is used
>>> to indicate the way to reference the variable in memory using the string
>>> value from my array varnames that was queried from the file
>>> end do
>>>
>>> If there was a way to reference the variable from memory, one could do
>>> all the processing on an arbitrary variable without ever knowing it's
>>> name. This would be a nice feature to have.
>>>
>>> Jonathan
>>>
>>>
>>>
>>> Dave Allured wrote:
>>>
>>>
>>>> Jonathan,
>>>>
>>>> Do you need to have all the variables in memory at the same time, or
>>>> can you just process them one at a time from the input file to the
>>>> output file(s)? The second has a much easier answer than the first.
>>>> A little more information about the nature of your application is
>>>> needed.
>>>>
>>>> Dave Allured
>>>> CU/CIRES Climate Diagnostics Center (CDC)
>>>> http://cires.colorado.edu/science/centers/cdc/
>>>> NOAA/ESRL/PSD, Climate Analysis Branch (CAB)
>>>> http://www.cdc.noaa.gov/psd1/
>>>>
>>>> Jonathan Vigh wrote:
>>>>
>>>>
>>>>
>>>>> Greetings NCL'ers,
>>>>> I'd like to read in all the variables from a netCDF file in a
>>>>> general way because there are 177 of them. I understand how to query the
>>>>> file and get a list of variable names, and I even understand how to
>>>>> reference the variable using:
>>>>> fin->$varnames$
>>>>>
>>>>> What I am confused about is what to assign this to. I'd like to avoid
>>>>> having to type out 177 lines to read in each variable if possible. But
>>>>> if I didn't even know the variable names in advance, I'd still like to
>>>>> know how to do this.
>>>>>
>>>>> Maybe I've missed something really basic, but I couldn't seem to find
>>>>> anything from the documentation. If anyone has an idea, I'd be most
>>>>> grateful. It would be nice if NCL could simply make all the variables in
>>>>> a netCDF file available without having to go through this step, so maybe
>>>>> this could be the prototype for a new function that does this.
>>>>>
>>>>> Thanks,
>>>>> Jonathan
>>>>>
>>>>>
>>>>> begin
>>>>> processed_filename = systemfunc("cd " + processed_data_directory + ";
>>>>> ls " + stormid + "*.nc")
>>>>> fin = addfile(processed_data_directory + processed_filename,"r") ;
>>>>> open output netCDF file
>>>>>
>>>>> ; query the file to see what variables it contains and what the sizes of
>>>>> the file dimensions are
>>>>> varnames = getfilevarnames(fin) ; Returns an array of file
>>>>> variable names in the file
>>>>> filedimsizes = getfiledimsizes(fin) ; Returns a list of the
>>>>> sizes of all the dimensions in the file.
>>>>>
>>>>> ; query all the named variables in the file
>>>>> vartypes = getfilevartypes(fin,varnames) ; Returns the type of
>>>>> each variable name listed. A missing value is returned for any variable
>>>>> name that doesn't exist in the file.
>>>>>
>>>>>
>>>>> ; loop through all the variables, query them, and read them in
>>>>> nvars = dimsizes(varnames)
>>>>> do ivar = 0, nvars-1
>>>>> print("Reading variable ivar = "+ivar+" "+vartypes(ivar)+"
>>>>> "+varnames(ivar))
>>>>> ???? = fin->$varnames(ivar)$ ; I understand how to
>>>>> read in a variable from a file if I don't know the variable name in
>>>>> advance, but how can I assign it to a variable without typing the
>>>>> variable's name?
>>>>> end do
>>>>>
>>>>> end
>>>>> _______________________________________________
>>>>> ncl-talk mailing list
>>>>> List instructions, subscriber options, unsubscribe:
>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> ncl-talk mailing list
>>>> List instructions, subscriber options, unsubscribe:
>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>
>>>>
>>>>
>>> _______________________________________________
>>> ncl-talk mailing list
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>>
>>>
>>
>>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>

;********************************************************
; main.ncl
;
; This program demonstrates a general method to read in the contents
; of a netCDF file without knowing the variable names in advance.
;
;********************************************************
;load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
;load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
;load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/shea_util.ncl"
;load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
load "./module_read_netcdf.ncl"
load "./module_dynamically_generated_part.ncl"
; load your next "global" module
; then load any modules containing procedures or functions
;********************************************************
begin

  print(" Now in the main calling program.")

  print("")
  print("The file's variables are:")
  print("")

  print(" "+varnames)

; now do fun stuff with your data . . .

end

;********************************************************
; module_dynamically_generated_part.ncl
; This module was generated dynamically by NCL to
; read in all the variables from a file since we may
; not know the variable names before run-time.
;********************************************************
var3 = fin->var3
var2 = fin->var2
var1 = fin->var1

;********************************************************
; module_read_netCDF.ncl
;********************************************************
; This module contains a function to read all the variables
; from a netCDF file without knowing the file contents in
; advance. It opens the file, queries it to determine
; which variables are present, and then reads them in..
;
; In order to make them available, all of this is done
; before any other part of the subsequent program runs.
; i.e. the variables will be global and therefore
; accessible to all functions and procedures in the
; main program which loads this module.
;
; Author: Jonathan Vigh, Colorado State University, 02/26/2009
; (with a helpful contribution by Saji N. Hameed, University of Hawaii)
;
;********************************************************
;
; Note that we do not put the contents of this module within 'begin' and 'end' statements.
; This means that this everything in this module will be global (accessible to all other
; functions and procedures in subsequent modules and the main program which loads this
; module.

; the filename can either be given here or as a command-line argument
  ncdf_filename = "foo.nc"

;===================================================================
; Read the processed data file which contains the best track and VDM data for this storm
;===================================================================
  print("")
  print("Now opening the netCDF data file: "+ncdf_filename)
  fin = addfile(ncdf_filename,"r") ; open output netCDF file

; retrieve the global attributes from the file
  fatt_names = getvaratts(fin)
  if(.not.all(ismissing(fatt_names))) then
     print("")
     print("The file's global attributes are: ")
     print("")
     
     do i = 0,dimsizes(fatt_names)-1
        print(fatt_names(i)+" = "+fin@$fatt_names(i)$)
     end do
  
  end if

; query the file to see what variables it contains and what the sizes of the file dimensions are
  varnames = getfilevarnames(fin) ; Returns an array of file variable names in the file
  filedimsizes = getfiledimsizes(fin) ; Returns a list of the sizes of all the dimensions in the file.

; query all the named variables in the file
  vartypes = getfilevartypes(fin,varnames) ; Returns the type of each variable name listed. A missing value is returned for any variable name that doesn't exist in the file.

  nvars = dimsizes(varnames)

; Now use NCL to generate some code that we will then load and use to read in all the variables
; we will generate the lines of the module sequentially and attach them as strings to a boolean variable
; then we'll write all the strings out using asciiwrite.

  cr = inttochar(10)

  module_filename = "module_dynamically_generated_part.ncl"

  module_contents = ";********************************************************" + cr \
                  + "; " + module_filename + cr \
                  + "; This module was generated dynamically by NCL to" + cr \
                  + "; read in all the variables from a file since we may" + cr \
                  + "; not know the variable names before run-time." + cr \
                  + ";********************************************************"

  do ivar = 0, nvars - 1
     module_contents = module_contents + cr + varnames(ivar)+" = fin->"+varnames(ivar)
  end do

  if(isfilepresent(module_filename))
    system("/bin/rm -f " + module_filename) ; remove a pre-existing file
  end if
 
  asciiwrite(module_filename,module_contents)
  
; The dynamically-generated module will then be loaded immediately after this module has been loaded in the calling program.
; Both modules must be loaded before the 'main' program begins (i.e. before the 'begin' statement).
 

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk