NCEP RTMA grib2 files and proposed changes to the GRIB2 interface

From: David Brown <dbrown_at_nyahnyahspammersnyahnyah>
Date: Wed Mar 27 2013 - 14:37:16 MDT

Hi Winston and all users of the NCL GRIB2 file reader,

I found out the distinction between the 2 GRIB records in the NCEP RTMA files, and have made some experimental changes to the GRIB2 reader that accounts for them. There is an octet in the GRIB2 Product Description Section that specifies "Type of Generating Process". in most GRIB2 files I have encountered this is set to a single value for every record in the file, most often to indicate "Forecast" as the value. Even where there are different values, it is not the only distinguishing difference between variables. In this file, however, is is the sole distinguishing difference in the metadata between the two records: one record is set to "Analysis" while the other is set to "Analysis error" . Adding an attribute for this octet allows an NCL user to distinguish these records:

      float TMP_P0_L103_GLC0 ( ygrid_0, xgrid_0 )
         center : US National Weather Service - NCEP (WMC)
         generating_process_type : Analysis
         production_status : Operational products
         long_name : Temperature
         units : K
         _FillValue : 1e+20
         coordinates : gridlat_0 gridlon_0
         grid_type : Lambert Conformal can be secant or tangent, conical or bipolar
         parameter_discipline_and_category : Meteorological products, Temperature
         parameter_template_discipline_category_number : ( 0, 0, 0, 0 )
         level_type : Specified height level above ground (m)
         level : 2
         forecast_time : 0
         forecast_time_units : hours
         initial_time : 07/01/2011 (00:00)

      float TMP_P0_L103_GLC0_1 ( ygrid_0, xgrid_0 )
         center : US National Weather Service - NCEP (WMC)
         generating_process_type : Analysis error
         production_status : Operational products
         long_name : Temperature
         units : K
         _FillValue : 1e+20
         coordinates : gridlat_0 gridlon_0
         grid_type : Lambert Conformal can be secant or tangent, conical or bipolar
         parameter_discipline_and_category : Meteorological products, Temperature
         parameter_template_discipline_category_number : ( 0, 0, 0, 0 )
         level_type : Specified height level above ground (m)
         level : 2
         forecast_time : 0
         forecast_time_units : hours
         initial_time : 07/01/2011 (00:00)

However, the cost of this solution is more attribute bloat for all the files where this octet is constant for every variable. There are other examples of generally file-wide constant attributes: e.g. "center" and "production_status". There are many other specifiers in the GRIB2 spec that could occasionally be used to distinguish otherwise identically described records. It doesn't seem feasible to anticipate them all in advance, but on the other hand, it would be nice to develop a system where they could be incorporated easily when they show up.

I am proposing to lessen the attribute bloat by turning file-wide constant attributes into global attributes. Neither the GRIB1 or GRIB2 reader have ever made use of global attributes but there is nothing that actually stands in the way of adapting them other than the fact that it might lead to a minor amount of backwards-incompatibility. The way it would work is that as long as all variables have the same value for an attribute, it becomes a global attribute. If any single variable has a different value, then it remains as a variable attribute. I can see this could be problematic for users who expect to find important metadata like initial_time, and forecast_time as variable attributes, and therefore, at least at first, coordinate-related attributes would remain as variable attributes.
But eventually a file option could allow a user to choose to force all attributes to be retained as variable attributes.

This may also allow for new attributes that relate to the file as a whole -- attributes that would be too wordy to repeat for each variable but which would be acceptable and possibly useful if they only appear once per file.

Note in the case above, the variable names are only distinguished by an '_1' following the second variable name. I don't necessarily like this, but I also do not want to cause the distinction between generating process types to result in another suffix that has to be applied to all variable names. The '_x' is only applied when the names cannot be distinguished any other way.

Feedback from any affected users is welcomed and appreciated.
 -dave

On Mar 6, 2013, at 1:27 PM, David Brown <dbrown@ucar.edu> wrote:

> Yes, that is correct. At the moment NCL cannot find anything to distinguish these 2 records, and eliminates one of them as a "duplicate" record.
> Of course the actual data is quite different.
> I hope to modify the code to at least give two variables with the second one having an '_1' attached to it.
> -dave
>
> On Mar 6, 2013, at 1:21 PM, Winston Hao wrote:
>
>> Hi Gerry,
>>
>> I plotted the variable TMP_L103_GLC0 and found out it is
>> the analysis uncertainty field. My guess is the 1st record, the
>> temperature record, is treated as a duplicate record. But I am
>> not sure.
>>
>> Winston
>>
>>
>> Winston Hao
>> New York State Dept of Environmental Conservation
>> 625 Broadway, 9th Floor, Albany, NY 12233-3259
>> whao@dec.state.ny.us (518)402-8384 (Work) (518)402-9035 (Fax)
>>
>> On Wed, 6 Mar 2013, Gerry Creager - NOAA Affiliate wrote:
>>
>>> I've not looked in depth but a quick pass with ncl_filedump shows a
>>> variable TMP_P0_L103_GLC0 (Temperature, K), and grid-related variables for
>>> rotation, lat and lon. Nothing for analysis uncertainty that I could find.
>>>
>>> gerry
>>>
>>>
>>> On Wed, Mar 6, 2013 at 1:08 PM, David Brown <dbrown@ucar.edu> wrote:
>>>
>>>> Hi Winston,
>>>>
>>>> I will look into this. Thanks for the report.
>>>> -dave
>>>>
>>>> On Mar 6, 2013, at 11:07 AM, Winston Hao wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> I try to plot NCEP Real-Time Mesoscale Analysis (RTMA) hourly
>>>>> 2-m temperature field using ncl, but I could not read those grib2 files
>>>>> properly. I have uploaded an example file called
>>>> LTIA98_KWBR_201107010000
>>>>> to ftp.cgd.ucar.edu:/incoming The same file can be downloaded directly
>>>>> from NCDC RTMA archive via wget. The link is
>>>>>
>>>> http://nomads.ncdc.noaa.gov/data/ndgd/201107/20110701/LTIA98_KWBR_201107010000
>>>>> The file size is about 8 megabytes.
>>>>>
>>>>> The file should have two records, one for the temperature analysis
>>>> and the
>>>>> other for the analysis uncertainty. Wgrib2 utility confirms that:
>>>>>
>>>>> 1:0:d=2011070100:TMP:2 m above ground:anl:
>>>>> 2:4799894:d=2011070100:TMP:2 m above ground:anl:analysis/forecast error
>>>>>
>>>>> However, in my ncl script, or from ncl_filedump, or from
>>>> ncl_convert2nc,
>>>>> I can only see one record, the 2nd record. My work-around is to use
>>>> wgrib2
>>>>> to extract the record I want and then use ncl to process it.
>>>>>
>>>>> I am wondering if there are ways to access the 1st record in ncl.
>>>>>
>>>>> I am using ncl 6.1.2 (not OPeNDAP-enabled pre-compiled binary)
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>> Winston
>>>>>
>>>>>
>>>>> Winston Hao
>>>>> New York State Dept of Environmental Conservation
>>>>> 625 Broadway, 9th Floor, Albany, NY 12233-3259
>>>>> whao@dec.state.ny.us (518)402-8384 (Work) (518)402-9035 (Fax)
>>>>>
>>>>> _______________________________________________
>>>>> ncl-talk mailing list
>>>>> List instructions, subscriber options, unsubscribe:
>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>
>>>> _______________________________________________
>>>> ncl-talk mailing list
>>>> List instructions, subscriber options, unsubscribe:
>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>
>>>
>>>
>>>
>>> --
>>> Gerry Creager
>>> NSSL/CIMMS
>>> 405.325.6731
>>> ++++++++++++++++++++++
>>> „Big whirls have little whirls,
>>> That feed on their velocity;
>>> And little whirls have lesser whirls,
>>> And so on to viscosity.‰
>>> Lewis Fry Richardson (1881-1953)
>>>
>>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Wed Mar 27 14:37:27 2013

This archive was generated by hypermail 2.1.8 : Tue Apr 02 2013 - 21:23:48 MDT