Re: NCEP CSFR grb2 errors

From: David Brown <dbrown_at_nyahnyahspammersnyahnyah>
Date: Wed Jul 20 2011 - 15:43:07 MDT

Hi Allison,
Just to add a bit to Gus and Dennis' comments, NCL 5.1.1 did not recognize as many types of statistical processing as the current version does, and would probably be tripped up eventually by the statistical processing type "Average of forecast averages, each of an equal specified duration, with 6-hour intervals between forecasts". So I definitely agree that you should try a newer version of NCL. I have not had a chance to look at the files in order to guess at the significance of the 17th file, but the error you got on the 16th file indicates that your version of NCL is quite confused at that point: the 1.3.table is very simple and has not changed since the original implementation of the NCL GRIB2 reader.
 -dave

On Jul 20, 2011, at 2:50 PM, Gus Correa wrote:

> Hi Allison
>
> Aha!
> I am glad to know I am not the only one to suffer
> with name de-mangling of GRIB file variable names!
> That is voodoo for sure ...
>
> Yes, by all means, you should upgrade to NCL 6.0.0,
> which has many improvements over the 5.X.Y series.
> I also followed Dennis wise advice,
> and upgraded from 5.1.1 a month ago.
>
> In any case, it won't hurt if you try ncl_convert2nc or ncl_filedump
> on the 17th and 1st files and compare the results, just to see
> if the albedo really disappeared or became reflectivity.
>
> I don't know anything about GDAS, but another possibility that comes to
> mind is whether your 17th file, guess, covers Antarctica in the winter
> darkness, or perhaps the satellite was offline during that period,
> and to save disk space, GRIB decided to skip the albedo
> variable altogether, which was a bunch of missing values anyway.
>
> Gus Correa
>
>
> Allison Steiner wrote:
>> hi Gus--
>>
>> Thanks for the suggestion - I also noticed that the NCEP files change the variable names, and did have a string match in my original script to avoid that problem, like this:
>>
>> ; file loop
>> do nf=0, nfil-1
>> f = addfile(fili(nf),"r") ; grib
>> names = getfilevarnames(f)
>> dimname = dimsizes(names)
>>
>> ; find the right variable: name changes with height and file
>> do i = 0,dimname-1
>> strsub = isStrSubset(names(i),"ALBDO")
>> if(strsub) then
>> varname = names(i)
>> end if
>> end do
>>
>> ncepall(nf,:,:)=f->$varname$
>> delete(f)
>> delete(names)
>> delete(dimname)
>> delete(varname)
>> delete(strsub)
>> end do
>>
>>
>> But I was still getting the crash on the 17th file (independent of what set of files I was using). Strange that yours is on the 5th. Agreed, it must be some sort of GRIB voodoo.
>>
>> Thanks for the suggestion; I'm hoping the version upgrade that Dennis suggested will do the trick!
>>
>> Allison
>>
>>
>> On Jul 20, 2011, at 4:02 PM, Gus Correa wrote:
>>
>>> Hi Allison, Dennis
>>>
>>> Given similar mishaps I had with sequences of NCEP grib1 files,
>>> I would guess the albedo variable may be there, perhaps under a slightly
>>> different name, which I believe is automatically generated by the
>>> NCL libraries, based on the structure and coordinates
>>> of each grib file (which doesn't seem to be constant or stable
>>> over a whole decade or many months).
>>>
>>> Have you tried to convert the 17th file in your decade
>>> to netCDF using ncl_convert2nc,
>>> then search the resulting netCDF file header (obtained via ncdump)
>>> for a substring of your variable name (say, 'ALBDO')?
>>> Perhaps compare also with the corresponding results from the
>>> 1st or from the 16th file in your decade?
>>>
>>> Somehow I always had problems with mangled/modified variable names
>>> in the 5th month, whereas Allison seems to stumble on the 17th.
>>> Does grib hate prime numbers, or does it have a taste for
>>> numerology perhaps? :)
>>> Oh well, why the heck do they use GRIB after all?!
>>>
>>> I hope this helps,
>>> Gus Correa
>>>
>>>
>>> Dennis Shea wrote:
>>>> Hello
>>>>
>>>> v5.1.1 is more than 2 years old. The builtin GRIB2 tables were not
>>>> available at that time. The current NCL version (6.0.0) has no problem
>>>> reading the file(s). You'll have to
>>>>
>>>> %> ncl_filedump flxf03.gdas.197901.grb2 -itime | less
>>>>
>>>> Copyright (C) 1995-2011 - All Rights Reserved
>>>> University Corporation for Atmospheric Research
>>>> NCAR Command Language Version 6.0.0
>>>> The use of this software is governed by a License Agreement.
>>>> See http://www.ncl.ucar.edu/ for more details.
>>>>
>>>> Variable: f
>>>> Type: file
>>>> filename: flxf03.gdas.197901
>>>> path: flxf03.gdas.197901.grb2
>>>> file global attributes:
>>>> dimensions:
>>>> initial_time0_hours = 1
>>>> lat_0 = 576
>>>> lon_0 = 1152
>>>> lv_DBLL0 = 4
>>>> variables:
>>>> float TMP_P8_L1_GGA0_avg744h ( initial_time0_hours, lat_0, lon_0 )
>>>> center : US National Weather Service - NCEP (WMC)
>>>> production_status : Operational products
>>>> long_name : Temperature
>>>> units : K
>>>> _FillValue : 1e+20
>>>> grid_type : Gaussian latitude/longitude
>>>> parameter_discipline_and_category : Meteorological
>>>> products, Temperature
>>>> parameter_template_discipline_category_number : ( 8, 0,
>>>> 0, 0 )
>>>> level_type : Ground or water surface
>>>> level : 0
>>>> type_of_statistical_processing : Average
>>>> statistical_process_duration : 744 hours (ending at
>>> forecast time)
>>>> forecast_time : 744
>>>> forecast_time_units : hours
>>>>
>>>> [***SNIP***]
>>>> float ALBDO_P8_L1_GGA0 ( initial_time0_hours, lat_0, lon_0 )
>>>> center : US National Weather Service - NCEP (WMC)
>>>> production_status : Operational products
>>>> long_name : Albedo
>>>> units : %
>>>> _FillValue : 1e+20
>>>> grid_type : Gaussian latitude/longitude
>>>> parameter_discipline_and_category : Meteorological
>>>> products, Physical atmospheric properties
>>>> parameter_template_discipline_category_number : ( 8, 0,
>>>> 19, 1 )
>>>> level_type : Ground or water surface
>>>> level : 0
>>>> type_of_statistical_processing : Average of forecast
>>>> averages, each of an equal specified duration, with 6-hour
>>>> intervals between forecasts.
>>>> statistical_process_duration : 3 hours (beginnng at
>>> forecast time)
>>>> number_in_average : 124
>>>> forecast_time : 0
>>>> forecast_time_units : hours
>>>> [***SNIP***]
>>>>
>>>> ================================================
>>>>
>>>> D
>>>>
>>> Allison Steiner wrote:
>>>> Hello--
>>>>
>>>> I'm trying to process the NCEP CSFR GRB2 temperature output for
>>>> 1979-2009 and am running into some problems.
>>>>
>>>> The attached script opens the files in a loop and tries to read in a
>>>> specified variable (ALBDO_P8_L1_GGA0) from each file. The script runs
>>>> for the first fifteen months with the usual NCL grib warnings, e.g.:
>>>> warning: Entry (205) not found in code table file
>>>> /usr/share/ncarg/grib2_codetables/ncep/4/4.10.table
>>>>
>>>> However, when it reaches the 16th file,the grib error changes slightly to
>>>>
>>>> warning: NclGRIB2: codetable file
>>>> "/usr/share/ncarg/grib2_codetables/ncep/1/1.3.table" not a valid GRIB2
>>>> code table.
>>>>
>>>> It then crashes on the subsequent file (the 17th) with the following error:
>>>> fatal: NclGRIB2: codetable directory "/usr/share/ncarg/grib2_codetables"
>>>> invalid
>>>> fatal:Could not open (../Ncep/flxf03.gdas.198006.grb2)
>>>> fatal:Either file (f) isn't defined or variable (ALBDO_P8_L1_GGA0) is
>>>> not a variable in the file
>>>> fatal:Execute: Error occurred at or near line 19 in file testncep2.ncl
>>>>
>>>> I would suspect this was something with the libraries, but the script
>>>> runs for multiple months before getting this error. Also, the crash
>>>> always occurs on the 17th file, even if I change the decade of data I'm
>>>> processing, indicating that the error is not due to any one specific
>>>> file. I'm running NCL v. 5.1.1. Two years of monthly data are loaded
>>>> to (ftp.cgd.ucar.edu <http://ftp.cgd.ucar.edu/>) with the beginning file
>>>> extension of flxf03.gdas.YYYYMM.grb2.
>>>>
>>>> Any ideas?
>>>>
>>>> Thank you,
>>>>
>>>> Allison
>>>>
>>>>
>>>> ;************************************
>>>> ; Script to read in NCEP CSFR data 1979-2009
>>>> ;************************************
>>>> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
>>>> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
>>>> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
>>>>
>>>> begin
>>>>
>>>> ; list all files
>>>> fili = systemfunc("ls ../Ncep/*.grb2")
>>>> nfil = dimsizes(fili)
>>>>
>>>>
>>>> ; file loop
>>>> do nf=0, nfil-1
>>>> print(nf+" "+fili(nf))
>>>> f = addfile(fili(nf),"r") ; grib
>>>> ntest=f->ALBDO_P8_L1_GGA0
>>>>
>>>> printVarSummary(ntest)
>>>>
>>>> delete(f)
>>>> delete(ntest)
>>>> end do
>>>>
>>>>
>>>> end
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> _______________________________________________
>>>> ncl-talk mailing list
>>>> List instructions, subscriber options, unsubscribe:
>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>> _______________________________________________
>>> ncl-talk mailing list
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>>
>>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Wed Jul 20 15:43:13 2011

This archive was generated by hypermail 2.1.8 : Fri Jul 29 2011 - 08:44:18 MDT