Re: ncl_convert2nc and cdo

From: David B. Reusch <dreusch_at_nyahnyahspammersnyahnyah>
Date: Wed May 02 2012 - 18:01:47 MDT

OK, I've been doing some more investigating. Apologies if I'm missing
something obvious that says this isn't NCL...

The situation gets murky once I have more than two records from the
original ERA-Interim grib file. This is equivalent to hitting the first
forecast, second day since these files consist of daily 6h and 12h
forecasts. I believe NCL is handling this by adding dimension
initial_time0_hours to the dimension forecast_time that was sufficient
for the first two records.

If I then either (1) use ncrename (or perhaps any other NCO command, I
haven't checked) or (2) use the ncl_convert2nc -U/-u flags to rename
initial_time0_hours to time, then CDO starts reporting a new generic
grid in addition to the expected gaussian lat/lon grid. So it looks
like something in the processing of the forecast dimension is leading to
CDO thinking there's a generic grid in the file.

Per NCO support, the idea of a grid is purely semantic and it's
impossible that ncrename is changing the file. I don't know enough to
say boo about grids being real or not, I just know something like a
"grid" exists in these files and it's breaking CDO tools. If I create
the netcdf file using CDO instead of ncl_convert2nc, these problems
don't come up (CDO uses just a single time dimension, not the two that
ncl_convert2nc creates). That is actually what I'm doing
"operationally" to avoid this problem, I just can't help thinking
there's something that can be fixed so that I can use ncl_convert2nc all
the time.

I don't know if CDO is misreading these files or not (perhaps griddes
has a bug), but I don't know any other way to get this information. (Of
course, I wasn't the least bit concerned with grids before this problem
came up!) I also don't know whether CDO is the only program out there
that thinks about "grids" in this manner.

In short, I'm not sure what my options are at this point. Is
ncl_convert2nc doing something "odd"? Can I tell ncl_convert2nc to just
use one time dimension? Is NCL out of the picture and this is all CDO's
fault?

I've created more files in /glade/user/dbr/fc/ncl_nco_test on
bluefire. grb files were created with:
wgrib input_fc.grb | head -n 4 | wgrib -i input_fc.grb -grib -o fc_4r.grb
ncl_convert2nc created fc_xr.nc from these grb files.
After copying fc_xr.nc to fc_xr_ren.nc, I used " ncrename -d
initial_time0_hours,time -v initial_time0_hours,time fc_3r_ren.nc" on
the files that include initial_time0_hours (3 and 4).
Output from "cdo griddes" attached below.

All constructive thoughts/suggestions welcome.

Thanks for your patience,
Dave

cdo griddes on fc_3r.nc (yval's omitted):
#
# gridID 0
#
gridtype = gaussian
gridsize = 131072
xname = g4_lon_3
xlongname = longitude
xunits = degrees_east
yname = g4_lat_2
ylongname = latitude
yunits = degrees_north
xsize = 512
ysize = 256
xfirst = 0
xinc = 0.703125

cdo griddes on fc_3r_ren.nc (yval's omitted):
#
# gridID 0
#
gridtype = generic
gridsize = 1
xname = x
xunits = degrees
yname = y
yunits = degrees
#
# gridID 1
#
gridtype = gaussian
gridsize = 131072
xname = g4_lon_3
xlongname = longitude
xunits = degrees_east
yname = g4_lat_2
ylongname = latitude
yunits = degrees_north
xsize = 512
ysize = 256
xfirst = 0
xinc = 0.703125

Dennis Shea said the following on 4/27/12 7:28 PM:
> Interesting: "confirm that CDO does in fact require the time
> coordinate to be named time"
>
> ---
> Back to the original thread: ncl_convert2nc when used with GRIB files
> creates 'almost' CF compliant netCDF
>
>
> For example, the nc file created by ncl_convert2nc
> that started the thread
>
> =======
> Using CF Checker Version 2.0.3
> Using Standard Name Table Version 19 (2012-03-22T10:58:54Z)
>
> WARNING (2.6.1): No 'Conventions' attribute present
>
> ------------------
> Checking variable: forecast_time1
> ------------------
>
> ------------------
> Checking variable: g4_lat_2
> ------------------
>
> ------------------
> Checking variable: g4_lon_3
> ------------------
>
> ------------------
> Checking variable: initial_time0
> ------------------
> ERROR (3.1): Invalid units: mm/dd/yyyy (hh:mm)
>
> ------------------
> Checking variable: initial_time0_encoded
> ------------------
> ERROR (3.1): Invalid units: yyyymmddhh.hh_frac
>
> ------------------
> Checking variable: STRD_GDS4_SFC
> ------------------
>
> ------------------
> Checking variable: initial_time0_hours
> ------------------
>
> ERRORS detected: 2
> WARNINGS given: 1
> INFORMATION messages: 0
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> The only thing not CF compliant are the units for the 'auxiliary'
> time variables: initial_time0 and initial_time0_encoded. The unlimited
> time variable is CF compliant.
>
>
> On 4/27/12 6:27 PM, Daran Rife wrote:
>> Hi Dave,
>>
>> You are correct about the CF doc, and that David's file appears to conform to CF. Apology for not studying that coordinate more carefully. I can confirm that CDO does in fact require the time coordinate to be named time. I am sure this was a design decision, since it would be more difficult to automatically sort out which dimension was time, since it could in principal be named anything.
>>
>> Regards,
>>
>>
>> Daran
>>
>>
>> Sent from my BlackBerry.
>>
>> ----- Original Message -----
>> From: David Brown [mailto:dbrown@ucar.edu]
>> Sent: Friday, April 27, 2012 10:41 PM
>> To: Daran Rife
>> Cc: ncl-talk@ucar.edu<ncl-talk@ucar.edu>
>> Subject: Re: ncl_convert2nc and cdo
>>
>> Hi Daran and David,
>> I do not agree that the file created by ncl_convert2nc is not CF-compliant with respect to the time coordinate. The section of the CF convention that you point to does not say that the time coordinate must be named "time" (although perhaps it is true that CDO requires this -- I don't know). What it does say is that "A time coordinate is identifiable from its units string alone." There is such a coordinate in the file:
>>
>> double initial_time0_hours(initial_time0_hours) ;
>> initial_time0_hours:units = "hours since 1800-01-01 00:00" ;
>> initial_time0_hours:long_name = "initial time" ;
>>
>> Since the "units" attribute conforms to the CF-conventions, that is sufficient to identify this variable as a time coordinate.
>>
>> If the coordinate must be renamed, at this point it is probably easier, as Daran says, to use NCO. You could directly output a NetCDF with renamed coordinate variable from the GRIB file using the template at http://www.ncl.ucar.edu/Applications/gribeff.shtml.
>>
>> it is not hard to rename the dimensions using NCL
>>
>> f!0 = "newname" ; where 0 is the index of the dimension names returned by getvardims(f)
>>
>> but there is not a good way of renaming the coordinate variable currently.
>> -dave
>>
>>
>>
>>
>> On Apr 27, 2012, at 1:25 PM, Daran Rife wrote:
>>
>>> Hi David,
>>>
>>>> From looking at your netCDF header, it is clear that your file
>>> is not Climate Forecast (CF) compliant.
>>>
>>> http://cf-pcmdi.llnl.gov/
>>>
>>> In particular, your file does not have the "time" coordinate
>>> defined according to the CF specifications.
>>>
>>> http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.htm
>>> l#time-coordinate
>>>
>>> CDO requires that your file conform to the CF convention,
>>> and must include the time coordinate. You have a "time"
>>> like coordinate, but these do not appear to conform to the CF
>>> convention. Note that your time coordinate must be named
>>> "time" as CDO will only recognize this. Coordinate (or
>>> record dimensions) can easily be renamed using netCDF
>>> Operators utility called ncrename.
>>>
>>> http://nco.sourceforge.net/nco.html#ncrename-netCDF-Renamer
>>>
>>> Syntax looks similar to this:
>>>
>>> % ncrename -d old_coord_name,new_coord_name.
>>>
>>> I do not recall whether NCL can be used to change the name of
>>> a coordinate variable, so I'll have to defer to one of the
>>> gurus for that.
>>>
>>> I hope this helps.
>>>
>>> Daran
>>>
>>> Daran Rife
>>> Head of Mesoscale Modeling
>>> GL Garrad Hassan
>>>
>>> 9665 Chesapeake Drive, Suite 435, San Diego, CA 92123, USA
>>> Tel: +1 303 271 1450 / Fax: +1 303 271 1450 / Mobile: +1 303 968 6821
>>> www.gl-garradhassan.com
>>>
>>> This email is issued on behalf of Garrad Hassan America, Inc. (trading
>>> as GL Garrad Hassan) and is subject to the GL Garrad Hassan email
>>> disclaimer (http://www.gl-garradhassan.com/emaildisclaimer). Garrad
>>> Hassan America, Inc. is registered in America No. 94-3402236 Registered
>>> Office: 45 Main Street, Suite 302, Peterborough, NH 03458, USA
>>>
>>> --
>>>
>>> Message: 9
>>> Date: Thu, 26 Apr 2012 16:04:07 -0600
>>> From: "David B. Reusch"<dreusch@ees.nmt.edu>
>>> Subject: ncl_convert2nc and cdo
>>> To: ncl-talk@ucar.edu
>>> Message-ID:<4F99C657.2000502@ees.nmt.edu>
>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>>
>>> Hello,
>>>
>>> I don't know if this is a NCL problem or a CDO problem, but since I can
>>> get it to work if I don't use the output of ncl_convert2nc, I thought
>>> I'd start with NCL. The problem is that I get the following when trying
>>>
>>> to use CDO on a netCDF file created by ncl_convert2nc:
>>>> cdo sellonlatbox,-90,-55,0,360 input.nc output.nc
>>>> cdf_get_vara_double : ncid = 65536 varid = 6
>>>>
>>>> Error (cdf_get_vara_double) : NetCDF: Index exceeds dimension bound
>>> If I apply cdo to the original grb file, I do not get this error.
>>>> cdo -f nc -t ecmwf sellonlatbox,-90,-55,0,360 input.grb output2.nc
>>>> cdo sellonlatbox: Processed 191889408 values from 1 variable over 1464
>>>> timesteps. ( 1.38s )
>>> The ncl_convert2nc command works without obvious errors:
>>>> ncl_convert2nc input.grb
>>>> Processing file: input.grb...
>>>> Copyright (C) 1995-2011 - All Rights Reserved
>>>> University Corporation for Atmospheric Research
>>>> NCAR Command Language Version 6.0.0
>>>> The use of this software is governed by a License Agreement.
>>>> See http://www.ncl.ucar.edu/ for more details.
>>>> Classic model NetCDF does not support string types, converting
>>>> initial_time0 to a character array
>>>> Dimension 'ncl_strlen_0' will be added
>>> The original file is a forecast file from ERA-Interim
>>> (ei.oper.fc.sfc.regn128sc.strd_175.1980). I copied it to input.grb to
>>> save typing!
>>>
>>> All of this is on bluefire. Files can be found in
>>> /glade/user/dbr/fc/ncl_cdo_test. See below for the ncdump output for
>>> the NCL-converted file.
>>>
>>> I've found alternate methods to get my needed processing done so this
>>> isn't an urgent problem. Mostly I'm just curious whether this is NCL or
>>>
>>> CDO (v1.5.3), and what the underlying cause might be.
>>>
>>> Thanks,
>>> Dave
>>>
>>> ncdump output:
>>> netcdf input {
>>> dimensions:
>>> initial_time0_hours = 732 ;
>>> forecast_time1 = 2 ;
>>> g4_lat_2 = 256 ;
>>> g4_lon_3 = 512 ;
>>> ncl_strlen_0 = 18 ;
>>> variables:
>>> char initial_time0(initial_time0_hours, ncl_strlen_0) ;
>>> initial_time0:NCL_converted_from_type = "string" ;
>>> initial_time0:units = "mm/dd/yyyy (hh:mm)" ;
>>> initial_time0:long_name = "Initial time of first
>>> record" ;
>>> int forecast_time1(forecast_time1) ;
>>> forecast_time1:units = "hours" ;
>>> forecast_time1:long_name = "Forecast offset from
>>> initial time" ;
>>> float g4_lon_3(g4_lon_3) ;
>>> g4_lon_3:La1 = 89.463f ;
>>> g4_lon_3:Lo1 = 0.f ;
>>> g4_lon_3:La2 = -89.463f ;
>>> g4_lon_3:Lo2 = 359.297f ;
>>> g4_lon_3:Di = 0.7031252f ;
>>> g4_lon_3:N = 128.f ;
>>> g4_lon_3:units = "degrees_east" ;
>>> g4_lon_3:GridType = "Gaussian Latitude/Longitude Grid"
>>> ;
>>> g4_lon_3:long_name = "longitude" ;
>>> float g4_lat_2(g4_lat_2) ;
>>> g4_lat_2:La1 = 89.463f ;
>>> g4_lat_2:Lo1 = 0.f ;
>>> g4_lat_2:La2 = -89.463f ;
>>> g4_lat_2:Lo2 = 359.297f ;
>>> g4_lat_2:Di = 0.7031252f ;
>>> g4_lat_2:N = 128.f ;
>>> g4_lat_2:units = "degrees_north" ;
>>> g4_lat_2:GridType = "Gaussian Latitude/Longitude Grid"
>>> ;
>>> g4_lat_2:long_name = "latitude" ;
>>> double initial_time0_encoded(initial_time0_hours) ;
>>> initial_time0_encoded:units = "yyyymmddhh.hh_frac" ;
>>> initial_time0_encoded:long_name = "initial time encoded
>>>
>>> as double" ;
>>> double initial_time0_hours(initial_time0_hours) ;
>>> initial_time0_hours:units = "hours since 1800-01-01
>>> 00:00" ;
>>> initial_time0_hours:long_name = "initial time" ;
>>> float STRD_GDS4_SFC(initial_time0_hours, forecast_time1,
>>> g4_lat_2, g4_lon_3) ;
>>> STRD_GDS4_SFC:parameter_number = 175 ;
>>> STRD_GDS4_SFC:parameter_table_version = 128 ;
>>> STRD_GDS4_SFC:gds_grid_type = 4 ;
>>> STRD_GDS4_SFC:level_indicator = 1 ;
>>> STRD_GDS4_SFC:_FillValue = 1.e+20f ;
>>> STRD_GDS4_SFC:units = "W m**-2 s" ;
>>> STRD_GDS4_SFC:long_name = "Surface thermal radiation
>>> downwards" ;
>>> STRD_GDS4_SFC:center = "European Center for
>>> Medium-Range Weather Forecasts (RSMC)" ;
>>>
>>> // global attributes:
>>> :creation_date = "Thu Apr 26 15:49:03 MDT 2012" ;
>>> :NCL_Version = "6.0.0" ;
>>> :system = "AIX be1005en 3 5 00CBF2314C00 AIX" ;
>>> :conventions = "None" ;
>>> :grib_source = "input.grb" ;
>>> :title = "NCL: convert-GRIB-to-netCDF" ;
>>> }
>>>
>>>
>>>
>>> _______________________________________________
>>> ncl-talk mailing list
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>

-- 
Associate Research Professor of Climatology
Dept of Earth and Environmental Science
MSEC 304; 801 Leroy Place
New Mexico Tech
Socorro, NM 87801

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Wed May 2 18:02:00 2012

This archive was generated by hypermail 2.1.8 : Thu May 03 2012 - 09:27:53 MDT