Re: simple question: adding a parameter to a WRF output file

I think Dennis' explanation is perfectly correct. I would just like
to add that adding any new variable to an existing large NetCDF
(3.x) file can be expensive because NetCDF puts all the metadata at
the beginning, unless the original creator has gone to the trouble of
reserving extra space (see setfileoption 'HeaderReserveSpace'). This
means that all the actual data following the header has to be moved
regardless of whether there is an unlimited dimension. Unlimited
dimensions do, of course, make the move more complicated and probably
even more time-consuming.
  -dave

On Apr 16, 2009, at 7:12 AM, Dennis Shea wrote:

> offline, I suggested that Gerry use the netCDF operator "ncks"
> =====
> You could create the "new diagnostic value" in NCL;
> create the netCDF; then, using the system command,
> invoke the ncks
>
> f = addfile ("wrf_input.nc" , "r")
>
> ... your code ..
>
> fout = addfile ("wrf_add.nc" , "c") ;
>
> fout->FOO = whatever
>
> delete(fout) ; don't think this is necessary
>
> system("ncks wrf_input.nc wrf_add.f")
>
> or vice versa
>
> =====
> This still might take some time even with "ncks".
> Why?????
>
> Well if the new variable being written is (say) FOO(Time,....)
> *and* the time dimension is unlimited, then the file must
> be rewritten. As I understand it, when the record dimension is
> unlimited the variables are interleaved.
> =====
> The following is from an email I sent some time ago.
> It is a conceptual 'pictograph' of what data on a
> netCDF files look like when written with and without
> an unlimited record dimension
> [usually "time"] .
>
> Note: my mailer sometime reformats my emails [frustrating!]
> so I'm not sure how the alignments will look when sent.
> ========================
>
> Let's say there are 2 variables
> a(time,lev,lat,lon),
> b(time,lat,lon)
> where time>1
>
> If the record dimension is not tagged as unlimited ,
> the data are written as a block write.
>
> ---------
> netCDF header stuff:
> variable/dimension names, dimension sizes, attributes,......
> ---------
> | | t=0 [lev,lat,lon]
> | | t=1
> | |
> | a |
> | |
> | |
> | | t=N
> ---------
> followed by
> ---------
> | | t=0 [lat,lon]
> | | t=1
> | |
> | b |
> | |
> | |
> | | t=N
> ---------
>
> In this case, adding a new variable "c" would be 'simple'.
> Just add (append) a block of data at the end. [Actually,
> there is more to it than that.]
>
> ================
>
> With an unlimited dimension the variables are interleaved with
> the record dimension [typically, "time"]
>
> ________
> t=0 | |
> | a | a(lev,lat,lon) at t=0
> | |
> | |
> --------
> | |
> | b | b(lat,lon) at t=0
> --------
> ________
> t=1 | |
> | a | a(lev,lat,lon) at t=1
> | |
> | |
> --------
> | |
> | b | b(lat,lon) at t=1
> --------
> :
> :
> :
> ________
> t=N | |
> | a | a(lev,lat,lon) at t=N
> | |
> | |
> --------
> | |
> | b | b(lat,lon) at t=N
> --------
>
> Now if you have one or more (say) records to add, the data
> can be appended to the current file because you are just adding
> additional records at the end.
>
> ________
> t=N+1 | |
> | a' | a'(lev,lat,lon) at t=N+1
> | |
> | |
> --------
> | |
> | b' | b'(lat,lon) at t=N+1
> --------
> ________
> t=N+2 | |
> | a' | a'(lev,lat,lon) at t=N+2
> | |
> | |
> --------
> | |
> | b' | b'(lat,lon) at t=N+2
> --------
> :
> etc
> :
>
> +++++++++++++++++++
> Now the issue with adding a completely new variable (say) "c(time,...)
> Since, the variables are interleaved, the variable "c" must be
> inserted
> at each time step. No way to do this without a file rewrite.
>
> D
>
>
>
>
>
> Gerry Creager wrote:
>> Thanks, Mary.
>>
>> I did try that last night, but the wrfout I'm working on is fairly
>> big,
>> and things were progressing so slowly I'd thought I had something
>> broken. I'm processing the data now to see how it'll look in
>> VAPOR, and
>> ncl_filedump tells me I likely have a couple of small tweaks to make
>> before this is all resolved, anyway.
>>
>> Appreciate the help!
>> Regards, Gerry
>>
>> Mary Haley wrote:
>>> Hi Gerry,
>>>
>>> You should be able to do this easily if you open the file with "w"
>>> instead of "r":
>>>
>>> a = addfile("wrfout.nc","w")
>>>
>>> ....some calculations and other stuff...
>>>
>>> a->diagnostics_var = some_local_var
>>>
>>> Of course, you'd want to replace "diagnostics_var" with whatever
>>> name
>>> you want on the file.
>>>
>>> --Mary
>>>
>>> On Apr 15, 2009, at 7:59 PM, Gerry Creager wrote:
>>>
>>>> Hello, again,
>>>>
>>>> I'm reading a WRF output file, operating on it, and creating a new
>>>> diagnostic value. I need to write it back into the wrfout as an
>>>> append.
>>>>
>>>> Is there a fairly simple method to accomplish this, or do I need
>>>> to read
>>>> in, then write out all the variables?
>>>>
>>>> Thanks, Gerry
>>>> --
>>>> Gerry Creager -- gerry.creager_at_tamu.edu
>>>> Texas Mesonet -- AATLT, Texas A&M University
>>>> Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983
>>>> Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX
>>>> 77843
>>>> _______________________________________________
>>>> ncl-talk mailing list
>>>> List instructions, subscriber options, unsubscribe:
>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Thu Apr 16 2009 - 11:33:07 MDT