Hi Jonathan,
Congratulations for solving the problem and much thanks for sharing
your new code with us...
Best wishes,
saji
--
* Jonathan Vigh <vigh_at_atmos.colostate.edu> [2009-02-26 22:04:03 +0000]:
> Hi all,
> After thinking some more, I realized a way to incorporate Saji's
> suggestion as a module that runs at the beginning of the program. This
> module opens the file and queries it to obtain the list of variables.
> Using that variable list, it dynamically generates another module which
> contains the lines to read them from the file. This module should be
> loaded immediately after the first one. Both modules are called from the
> main calling program outside of the main statement block, so all the
> variables read in from the file will be global and available to all
> functions and procedures throughout the program.
>
> Since all that is required is to load the two modules at the top of your
> program, I think this is a pretty unobtrusive way to read in the
> contents of a netCDF file with many or unknown variables.
>
> I've attached the module_read_netcdf.ncl code, the module that was
> generated dynamically (which will be overwritten at run-time), an
> example 'main.ncl', and a simple netCDF file for those who are curious
> to try this out.
>
> Thanks for the helpful suggestion!
>
> Jonathan
>
>
>
>
> Jonathan Vigh wrote:
>> Hi Saji,
>> Thanks for the suggestion - that is an innovative idea. I was
>> hoping to modify this idea a bit and write code to generate just the
>> lines of code I would need to read in each variable, write these out as
>> a module, and then load that module within my program in the spot I
>> need it - but unfortunately "load" only works if the module is loaded
>> outside of the block that is planning to use it, so I get an error when
>> I try to load it:
>> "input in flex scanner failed"
>>
>> So for now, I'm just going to use NCL to generate the 100+ lines to
>> reach each variable and cut and paste it into my program (and repeat
>> this if the variables in my netCDF files change).
>>
>> Hopefully new versions of NCL can include some of these nifty features,
>> like an "eval" function, a way to create variables without assignment,
>> and a way to reference a variable without knowing it's name before
>> runtime.
>>
>> Jonathan
>>
>>
>>
>>
>>
>> Saji N. Hameed wrote:
>>
>>> Hi Jonathan,
>>>
>>> As a follow up of my previous mail related to "eval", why don't you
>>> run ncl on your code twice? The first pass will create NCL code that
>>> will be run in the second pass.
>>>
>>> Here is my crude semi-pseudo-code
>>>
>>> Start of NCL file ----------
>>>
>>> _NCLCode="" ; declared at the outset. global variable??
>>>
>>> function cr()
>>> begin
>>> return(inttochar(10))
>>> end
>>>
>>> procedure add_msg(msg)
>>> begin
>>> _NCLCode=_NCLCode+cr+msg
>>> end
>>>
>>>
>>> begin
>>> do ivar = 0,nvar-1
>>> msg =varnames(ivar)+" = fin->$"+varnames(ivar)+"$"
>>> add_msg(msg)
>>> end do
>>>
>>> asciiwrite("runit.ncl",_NCLCode) system("ncl runit.ncl")
>>> end
>>>
>>>
>>>
>>> * Jonathan Vigh <vigh_at_atmos.colostate.edu> [2009-02-24 21:05:59 +0000]:
>>>
>>>
>>>> Hi David,
>>>> I'll likely be doing some conditional processing, where some
>>>> variables affect others, so I'd like to have them all in memory at
>>>> once. For this specific application, all the variables are quite
>>>> small - the entire netCDF file is <50 KB - so it won't be a problem
>>>> to hold them all in memory. I'm planning to read in about 200 of
>>>> these files and aggregate some of the 177 variables into several
>>>> multidimensional arrays for further data analysis. The variables
>>>> are a hodgepodge of arrays of strings, logicals, integers, floats,
>>>> etc. It's a mess. I've never had to work with such a disparate
>>>> collection of data before. The netCDF files I'm reading are
>>>> actually an intermediate (but necessary) step - the data were
>>>> originally read from several very nonstandard ascii file formats
>>>> and I wrote them to the netCDF files. I wrote out the 177 variables
>>>> using a general method, so that's why I was hoping to read in the
>>>> 177 variables in a general way. The trouble with creating the
>>>> variable without knowing it's name seems to be the main hitch at
>>>> the moment. Accessing it without knowing it's name is the other
>>>> hitch.
>>>>
>>>> So basically, I think need a "create_new_var" procedure (not
>>>> function) which would act like:
>>>> create_new_var( varname, vartype, vardimsize )
>>>>
>>>> One the variable is created in memory, I'd need a way to reference
>>>> it in order to assign data to it from the file.
>>>>
>>>> So my code to read in everything would be something like:
>>>> varnames = getfilevarnames(fin) ; Returns an array of file
>>>> variable names in the file
>>>> vartypes = getfilevartypes(fin,varnames) ; Returns the type
>>>> of each variable name listed. A missing value is returned for any
>>>> variable name that doesn't exist in the file.
>>>>
>>>> nvars = dimsizes(varnames)
>>>> do ivar = 0, nvars-1
>>>> create_new_var( varnames(ivar), vartypes(ivar),
>>>> getfilevardimsizes(fin,varnames(ivar)) )
>>>> #varnames(ivar)# = fin->$varnames(ivar)$ ; where # # is
>>>> used to indicate the way to reference the variable in memory using
>>>> the string value from my array varnames that was queried from the
>>>> file
>>>> end do
>>>>
>>>> If there was a way to reference the variable from memory, one could
>>>> do all the processing on an arbitrary variable without ever knowing
>>>> it's name. This would be a nice feature to have.
>>>>
>>>> Jonathan
>>>>
>>>>
>>>>
>>>> Dave Allured wrote:
>>>>
>>>>> Jonathan,
>>>>>
>>>>> Do you need to have all the variables in memory at the same time,
>>>>> or can you just process them one at a time from the input file to
>>>>> the output file(s)? The second has a much easier answer than the
>>>>> first. A little more information about the nature of your
>>>>> application is needed.
>>>>>
>>>>> Dave Allured
>>>>> CU/CIRES Climate Diagnostics Center (CDC)
>>>>> http://cires.colorado.edu/science/centers/cdc/
>>>>> NOAA/ESRL/PSD, Climate Analysis Branch (CAB)
>>>>> http://www.cdc.noaa.gov/psd1/
>>>>>
>>>>> Jonathan Vigh wrote:
>>>>>
>>>>>> Greetings NCL'ers,
>>>>>> I'd like to read in all the variables from a netCDF file in
>>>>>> a general way because there are 177 of them. I understand how
>>>>>> to query the file and get a list of variable names, and I even
>>>>>> understand how to reference the variable using:
>>>>>> fin->$varnames$
>>>>>>
>>>>>> What I am confused about is what to assign this to. I'd like to
>>>>>> avoid having to type out 177 lines to read in each variable if
>>>>>> possible. But if I didn't even know the variable names in
>>>>>> advance, I'd still like to know how to do this.
>>>>>>
>>>>>> Maybe I've missed something really basic, but I couldn't seem
>>>>>> to find anything from the documentation. If anyone has an idea,
>>>>>> I'd be most grateful. It would be nice if NCL could simply make
>>>>>> all the variables in a netCDF file available without having to
>>>>>> go through this step, so maybe this could be the prototype for
>>>>>> a new function that does this.
>>>>>>
>>>>>> Thanks,
>>>>>> Jonathan
>>>>>>
>>>>>>
>>>>>> begin
>>>>>> processed_filename = systemfunc("cd " +
>>>>>> processed_data_directory + "; ls " + stormid + "*.nc") fin
>>>>>> = addfile(processed_data_directory + processed_filename,"r")
>>>>>> ; open output netCDF file
>>>>>>
>>>>>> ; query the file to see what variables it contains and what the
>>>>>> sizes of the file dimensions are
>>>>>> varnames = getfilevarnames(fin) ; Returns an array of
>>>>>> file variable names in the file
>>>>>> filedimsizes = getfiledimsizes(fin) ; Returns a list
>>>>>> of the sizes of all the dimensions in the file.
>>>>>>
>>>>>> ; query all the named variables in the file
>>>>>> vartypes = getfilevartypes(fin,varnames) ; Returns the
>>>>>> type of each variable name listed. A missing value is returned
>>>>>> for any variable name that doesn't exist in the file.
>>>>>>
>>>>>>
>>>>>> ; loop through all the variables, query them, and read them in
>>>>>> nvars = dimsizes(varnames)
>>>>>> do ivar = 0, nvars-1
>>>>>> print("Reading variable ivar = "+ivar+"
>>>>>> "+vartypes(ivar)+" "+varnames(ivar))
>>>>>> ???? = fin->$varnames(ivar)$ ; I understand
>>>>>> how to read in a variable from a file if I don't know the
>>>>>> variable name in advance, but how can I assign it to a variable
>>>>>> without typing the variable's name?
>>>>>> end do
>>>>>>
>>>>>> end
>>>>>> _______________________________________________
>>>>>> ncl-talk mailing list
>>>>>> List instructions, subscriber options, unsubscribe:
>>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>>>
>>>>> _______________________________________________
>>>>> ncl-talk mailing list
>>>>> List instructions, subscriber options, unsubscribe:
>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>>
>>>> _______________________________________________
>>>> ncl-talk mailing list
>>>> List instructions, subscriber options, unsubscribe:
>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>
>>>>
>>>
>>
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>
> ;********************************************************
> ; main.ncl
> ;
> ; This program demonstrates a general method to read in the contents
> ; of a netCDF file without knowing the variable names in advance.
> ;
> ;********************************************************
> ;load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
> ;load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
> ;load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/shea_util.ncl"
> ;load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
> load "./module_read_netcdf.ncl"
> load "./module_dynamically_generated_part.ncl"
> ; load your next "global" module
> ; then load any modules containing procedures or functions
> ;********************************************************
> begin
>
> print(" Now in the main calling program.")
>
> print("")
> print("The file's variables are:")
> print("")
>
> print(" "+varnames)
>
> ; now do fun stuff with your data . . .
>
>
>
> end
>
> ;********************************************************
> ; module_dynamically_generated_part.ncl
> ; This module was generated dynamically by NCL to
> ; read in all the variables from a file since we may
> ; not know the variable names before run-time.
> ;********************************************************
> var3 = fin->var3
> var2 = fin->var2
> var1 = fin->var1
> ;********************************************************
> ; module_read_netCDF.ncl
> ;********************************************************
> ; This module contains a function to read all the variables
> ; from a netCDF file without knowing the file contents in
> ; advance. It opens the file, queries it to determine
> ; which variables are present, and then reads them in..
> ;
> ; In order to make them available, all of this is done
> ; before any other part of the subsequent program runs.
> ; i.e. the variables will be global and therefore
> ; accessible to all functions and procedures in the
> ; main program which loads this module.
> ;
> ; Author: Jonathan Vigh, Colorado State University, 02/26/2009
> ; (with a helpful contribution by Saji N. Hameed, University of Hawaii)
> ;
> ;********************************************************
> ;
> ; Note that we do not put the contents of this module within 'begin' and 'end' statements.
> ; This means that this everything in this module will be global (accessible to all other
> ; functions and procedures in subsequent modules and the main program which loads this
> ; module.
>
> ; the filename can either be given here or as a command-line argument
> ncdf_filename = "foo.nc"
>
>
> ;===================================================================
> ; Read the processed data file which contains the best track and VDM data for this storm
> ;===================================================================
> print("")
> print("Now opening the netCDF data file: "+ncdf_filename)
> fin = addfile(ncdf_filename,"r") ; open output netCDF file
>
> ; retrieve the global attributes from the file
> fatt_names = getvaratts(fin)
> if(.not.all(ismissing(fatt_names))) then
> print("")
> print("The file's global attributes are: ")
> print("")
>
> do i = 0,dimsizes(fatt_names)-1
> print(fatt_names(i)+" = "+fin@$fatt_names(i)$)
> end do
>
> end if
>
> ; query the file to see what variables it contains and what the sizes of the file dimensions are
> varnames = getfilevarnames(fin) ; Returns an array of file variable names in the file
> filedimsizes = getfiledimsizes(fin) ; Returns a list of the sizes of all the dimensions in the file.
>
> ; query all the named variables in the file
> vartypes = getfilevartypes(fin,varnames) ; Returns the type of each variable name listed. A missing value is returned for any variable name that doesn't exist in the file.
>
> nvars = dimsizes(varnames)
>
> ; Now use NCL to generate some code that we will then load and use to read in all the variables
> ; we will generate the lines of the module sequentially and attach them as strings to a boolean variable
> ; then we'll write all the strings out using asciiwrite.
>
> cr = inttochar(10)
>
> module_filename = "module_dynamically_generated_part.ncl"
>
> module_contents = ";********************************************************" + cr \
> + "; " + module_filename + cr \
> + "; This module was generated dynamically by NCL to" + cr \
> + "; read in all the variables from a file since we may" + cr \
> + "; not know the variable names before run-time." + cr \
> + ";********************************************************"
>
> do ivar = 0, nvars - 1
> module_contents = module_contents + cr + varnames(ivar)+" = fin->"+varnames(ivar)
> end do
>
> if(isfilepresent(module_filename))
> system("/bin/rm -f " + module_filename) ; remove a pre-existing file
> end if
>
> asciiwrite(module_filename,module_contents)
>
> ; The dynamically-generated module will then be loaded immediately after this module has been loaded in the calling program.
> ; Both modules must be loaded before the 'main' program begins (i.e. before the 'begin' statement).
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
--
Saji N. Hameed
APEC Climate Center +82 51 668 7470
National Pension Corporation Busan Building 12F
Yeonsan 2-dong, Yeonje-gu, BUSAN 611705 saji_at_apcc21.net
KOREA
_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Sun Mar 01 2009 - 18:58:10 MST
This archive was generated by hypermail 2.2.0 : Mon Mar 02 2009 - 16:45:42 MST