Re: ncl_convert2nc and cdo (Done)

From: David B. Reusch <dreusch_at_nyahnyahspammersnyahnyah>
Date: Mon May 07 2012 - 17:41:54 MDT

Thanks for the thoughts Dave, they got me thinking again about that
aspect as well. For this particular dataset I *think* I would be ok
with the "flattening", but I went ahead and played around with it some
more and found a way to preserve all the dimensions *and* have the CDO
commands work too. I still don't know if this is an "artifact" of
ncl_convert2nc or a "bug" in how CDO handles "grids", but I have things
going as I'd like now so I'm thinking it might not really matter.

For the record, it looks like the "spurious" generic grid that CDO was
showing being added (either by doing an ncrename or by using "-u/-U"
with ncl_convert2nc) was related to the var initial_time0_encoded (a cdo
selgrid,1 produced a file with just this var and time in it). Once that
var was out of the way (using a cdo selgrid,2 for the grid with
lat/lon/etc in it), everything else worked just fine.

Lastly, thanks Dennis for mentioning the -u/-U options. That is saving
me a step downstream. I'd noticed these in the help info but hadn't
ever tried them out before.

Cheers,
Dave

David Brown said the following on 5/2/12 6:34 PM:
> Hi Dave,
> I haven't had time to study your message in depth, but I do want to
> reply concerning the distinction between initial_time and
> forecast_time dimensions. I have to say that if CDO is compressing
> these dimensions into one, then, in my opinion, it is taking a
> shortcut that can lead to unresolvable situations. If more than one
> initial_time is present in the file, and there are multiple forecast
> times, you are likely to encounter cases where the same time is
> represented more than once: if the model is run every 12 hours, each
> run representing a new initial time, forecast time 18 (hours) for the
> first initial time represents the same time as forecast time 6 for the
> next model run. But of course the data is probably different. There
> really are 2 dimensions. I'm thinking that CDO makes it easy to
> extract subset of the records, and perhaps that is how it expects
> users to handle this difficulty.
> -dave
>
>
> On May 2, 2012, at 6:01 PM, David B. Reusch wrote:
>
>> OK, I've been doing some more investigating. Apologies if I'm
>> missing something obvious that says this isn't NCL...
>>
>> The situation gets murky once I have more than two records from the
>> original ERA-Interim grib file. This is equivalent to hitting the
>> first forecast, second day since these files consist of daily 6h and
>> 12h forecasts. I believe NCL is handling this by adding dimension
>> initial_time0_hours to the dimension forecast_time that was
>> sufficient for the first two records.
>>
>> If I then either (1) use ncrename (or perhaps any other NCO command,
>> I haven't checked) or (2) use the ncl_convert2nc -U/-u flags to
>> rename initial_time0_hours to time, then CDO starts reporting a new
>> generic grid in addition to the expected gaussian lat/lon grid. So
>> it looks like something in the processing of the forecast dimension
>> is leading to CDO thinking there's a generic grid in the file.
>>
>> Per NCO support, the idea of a grid is purely semantic and it's
>> impossible that ncrename is changing the file. I don't know enough
>> to say boo about grids being real or not, I just know something like
>> a "grid" exists in these files and it's breaking CDO tools. If I
>> create the netcdf file using CDO instead of ncl_convert2nc, these
>> problems don't come up (CDO uses just a single time dimension, not
>> the two that ncl_convert2nc creates). That is actually what I'm
>> doing "operationally" to avoid this problem, I just can't help
>> thinking there's something that can be fixed so that I can use
>> ncl_convert2nc all the time.
>>
>> I don't know if CDO is misreading these files or not (perhaps griddes
>> has a bug), but I don't know any other way to get this information.
>> (Of course, I wasn't the least bit concerned with grids before this
>> problem came up!) I also don't know whether CDO is the only program
>> out there that thinks about "grids" in this manner.
>>
>> In short, I'm not sure what my options are at this point. Is
>> ncl_convert2nc doing something "odd"? Can I tell ncl_convert2nc to
>> just use one time dimension? Is NCL out of the picture and this is
>> all CDO's fault?
>>
>> I've created more files in /glade/user/dbr/fc/ncl_nco_test on
>> bluefire. grb files were created with:
>> wgrib input_fc.grb | head -n 4 | wgrib -i input_fc.grb -grib -o fc_4r.grb
>> ncl_convert2nc created fc_xr.nc from these grb files.
>> After copying fc_xr.nc to fc_xr_ren.nc, I used " ncrename -d
>> initial_time0_hours,time -v initial_time0_hours,time fc_3r_ren.nc" on
>> the files that include initial_time0_hours (3 and 4).
>> Output from "cdo griddes" attached below.
>>
>> All constructive thoughts/suggestions welcome.
>>
>> Thanks for your patience,
>> Dave
>>
>> cdo griddes on fc_3r.nc (yval's omitted):
>> #
>> # gridID 0
>> #
>> gridtype = gaussian
>> gridsize = 131072
>> xname = g4_lon_3
>> xlongname = longitude
>> xunits = degrees_east
>> yname = g4_lat_2
>> ylongname = latitude
>> yunits = degrees_north
>> xsize = 512
>> ysize = 256
>> xfirst = 0
>> xinc = 0.703125
>>
>> cdo griddes on fc_3r_ren.nc (yval's omitted):
>> #
>> # gridID 0
>> #
>> gridtype = generic
>> gridsize = 1
>> xname = x
>> xunits = degrees
>> yname = y
>> yunits = degrees
>> #
>> # gridID 1
>> #
>> gridtype = gaussian
>> gridsize = 131072
>> xname = g4_lon_3
>> xlongname = longitude
>> xunits = degrees_east
>> yname = g4_lat_2
>> ylongname = latitude
>> yunits = degrees_north
>> xsize = 512
>> ysize = 256
>> xfirst = 0
>> xinc = 0.703125
>>
>> Dennis Shea said the following on 4/27/12 7:28 PM:
>>> Interesting: "confirm that CDO does in fact require the time
>>> coordinate to be named time"
>>>
>>> ---
>>> Back to the original thread: ncl_convert2nc when used with GRIB files
>>> creates 'almost' CF compliant netCDF
>>>
>>>
>>> For example, the nc file created by ncl_convert2nc
>>> that started the thread
>>>
>>> =======
>>> Using CF Checker Version 2.0.3
>>> Using Standard Name Table Version 19 (2012-03-22T10:58:54Z)
>>>
>>> WARNING (2.6.1): No 'Conventions' attribute present
>>>
>>> ------------------
>>> Checking variable: forecast_time1
>>> ------------------
>>>
>>> ------------------
>>> Checking variable: g4_lat_2
>>> ------------------
>>>
>>> ------------------
>>> Checking variable: g4_lon_3
>>> ------------------
>>>
>>> ------------------
>>> Checking variable: initial_time0
>>> ------------------
>>> ERROR (3.1): Invalid units: mm/dd/yyyy (hh:mm)
>>>
>>> ------------------
>>> Checking variable: initial_time0_encoded
>>> ------------------
>>> ERROR (3.1): Invalid units: yyyymmddhh.hh_frac
>>>
>>> ------------------
>>> Checking variable: STRD_GDS4_SFC
>>> ------------------
>>>
>>> ------------------
>>> Checking variable: initial_time0_hours
>>> ------------------
>>>
>>> ERRORS detected: 2
>>> WARNINGS given: 1
>>> INFORMATION messages: 0
>>>
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> The only thing not CF compliant are the units for the 'auxiliary'
>>> time variables: initial_time0 and initial_time0_encoded. The unlimited
>>> time variable is CF compliant.
>>>
>>>
>>> On 4/27/12 6:27 PM, Daran Rife wrote:
>>>> Hi Dave,
>>>>
>>>> You are correct about the CF doc, and that David's file appears to conform to CF. Apology for not studying that coordinate more carefully. I can confirm that CDO does in fact require the time coordinate to be named time. I am sure this was a design decision, since it would be more difficult to automatically sort out which dimension was time, since it could in principal be named anything.
>>>>
>>>> Regards,
>>>>
>>>>
>>>> Daran
>>>>
>>>>
>>>> Sent from my BlackBerry.
>>>>
>>>> ----- Original Message -----
>>>> From: David Brown [mailto:dbrown@ucar.edu]
>>>> Sent: Friday, April 27, 2012 10:41 PM
>>>> To: Daran Rife
>>>> Cc:ncl-talk@ucar.edu<ncl-talk@ucar.edu>
>>>> Subject: Re: ncl_convert2nc and cdo
>>>>
>>>> Hi Daran and David,
>>>> I do not agree that the file created by ncl_convert2nc is not CF-compliant with respect to the time coordinate. The section of the CF convention that you point to does not say that the time coordinate must be named "time" (although perhaps it is true that CDO requires this -- I don't know). What it does say is that "A time coordinate is identifiable from its units string alone." There is such a coordinate in the file:
>>>>
>>>> double initial_time0_hours(initial_time0_hours) ;
>>>> initial_time0_hours:units = "hours since 1800-01-01 00:00" ;
>>>> initial_time0_hours:long_name = "initial time" ;
>>>>
>>>> Since the "units" attribute conforms to the CF-conventions, that is sufficient to identify this variable as a time coordinate.
>>>>
>>>> If the coordinate must be renamed, at this point it is probably easier, as Daran says, to use NCO. You could directly output a NetCDF with renamed coordinate variable from the GRIB file using the template athttp://www.ncl.ucar.edu/Applications/gribeff.shtml.
>>>>
>>>> it is not hard to rename the dimensions using NCL
>>>>
>>>> f!0 = "newname" ; where 0 is the index of the dimension names returned by getvardims(f)
>>>>
>>>> but there is not a good way of renaming the coordinate variable currently.
>>>> -dave
>>>>
>>>>
>>>>
>>>>
>>>> On Apr 27, 2012, at 1:25 PM, Daran Rife wrote:
>>>>
>>>>> Hi David,
>>>>>
>>>>>> From looking at your netCDF header, it is clear that your file
>>>>> is not Climate Forecast (CF) compliant.
>>>>>
>>>>> http://cf-pcmdi.llnl.gov/
>>>>>
>>>>> In particular, your file does not have the "time" coordinate
>>>>> defined according to the CF specifications.
>>>>>
>>>>> http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.htm
>>>>> l#time-coordinate
>>>>>
>>>>> CDO requires that your file conform to the CF convention,
>>>>> and must include the time coordinate. You have a "time"
>>>>> like coordinate, but these do not appear to conform to the CF
>>>>> convention. Note that your time coordinate must be named
>>>>> "time" as CDO will only recognize this. Coordinate (or
>>>>> record dimensions) can easily be renamed using netCDF
>>>>> Operators utility called ncrename.
>>>>>
>>>>> http://nco.sourceforge.net/nco.html#ncrename-netCDF-Renamer
>>>>>
>>>>> Syntax looks similar to this:
>>>>>
>>>>> % ncrename -d old_coord_name,new_coord_name.
>>>>>
>>>>> I do not recall whether NCL can be used to change the name of
>>>>> a coordinate variable, so I'll have to defer to one of the
>>>>> gurus for that.
>>>>>
>>>>> I hope this helps.
>>>>>
>>>>> Daran
>>>>>
>>>>> Daran Rife
>>>>> Head of Mesoscale Modeling
>>>>> GL Garrad Hassan
>>>>>
>>>>> 9665 Chesapeake Drive, Suite 435, San Diego, CA 92123, USA
>>>>> Tel: +1 303 271 1450 / Fax: +1 303 271 1450 / Mobile: +1 303 968 6821
>>>>> www.gl-garradhassan.com
>>>>>
>>>>> This email is issued on behalf of Garrad Hassan America, Inc. (trading
>>>>> as GL Garrad Hassan) and is subject to the GL Garrad Hassan email
>>>>> disclaimer (http://www.gl-garradhassan.com/emaildisclaimer). Garrad
>>>>> Hassan America, Inc. is registered in America No. 94-3402236 Registered
>>>>> Office: 45 Main Street, Suite 302, Peterborough, NH 03458, USA
>>>>>
>>>>> --
>>>>>
>>>>> Message: 9
>>>>> Date: Thu, 26 Apr 2012 16:04:07 -0600
>>>>> From: "David B. Reusch"<dreusch@ees.nmt.edu>
>>>>> Subject: ncl_convert2nc and cdo
>>>>> To:ncl-talk@ucar.edu
>>>>> Message-ID:<4F99C657.2000502@ees.nmt.edu>
>>>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>>>>
>>>>> Hello,
>>>>>
>>>>> I don't know if this is a NCL problem or a CDO problem, but since I can
>>>>> get it to work if I don't use the output of ncl_convert2nc, I thought
>>>>> I'd start with NCL. The problem is that I get the following when trying
>>>>>
>>>>> to use CDO on a netCDF file created by ncl_convert2nc:
>>>>>> cdo sellonlatbox,-90,-55,0,360 input.nc output.nc
>>>>>> cdf_get_vara_double : ncid = 65536 varid = 6
>>>>>>
>>>>>> Error (cdf_get_vara_double) : NetCDF: Index exceeds dimension bound
>>>>> If I apply cdo to the original grb file, I do not get this error.
>>>>>> cdo -f nc -t ecmwf sellonlatbox,-90,-55,0,360 input.grb output2.nc
>>>>>> cdo sellonlatbox: Processed 191889408 values from 1 variable over 1464
>>>>>> timesteps. ( 1.38s )
>>>>> The ncl_convert2nc command works without obvious errors:
>>>>>> ncl_convert2nc input.grb
>>>>>> Processing file: input.grb...
>>>>>> Copyright (C) 1995-2011 - All Rights Reserved
>>>>>> University Corporation for Atmospheric Research
>>>>>> NCAR Command Language Version 6.0.0
>>>>>> The use of this software is governed by a License Agreement.
>>>>>> Seehttp://www.ncl.ucar.edu/ for more details.
>>>>>> Classic model NetCDF does not support string types, converting
>>>>>> initial_time0 to a character array
>>>>>> Dimension 'ncl_strlen_0' will be added
>>>>> The original file is a forecast file from ERA-Interim
>>>>> (ei.oper.fc.sfc.regn128sc.strd_175.1980). I copied it to input.grb to
>>>>> save typing!
>>>>>
>>>>> All of this is on bluefire. Files can be found in
>>>>> /glade/user/dbr/fc/ncl_cdo_test. See below for the ncdump output for
>>>>> the NCL-converted file.
>>>>>
>>>>> I've found alternate methods to get my needed processing done so this
>>>>> isn't an urgent problem. Mostly I'm just curious whether this is NCL or
>>>>>
>>>>> CDO (v1.5.3), and what the underlying cause might be.
>>>>>
>>>>> Thanks,
>>>>> Dave
>>>>>
>>>>> ncdump output:
>>>>> netcdf input {
>>>>> dimensions:
>>>>> initial_time0_hours = 732 ;
>>>>> forecast_time1 = 2 ;
>>>>> g4_lat_2 = 256 ;
>>>>> g4_lon_3 = 512 ;
>>>>> ncl_strlen_0 = 18 ;
>>>>> variables:
>>>>> char initial_time0(initial_time0_hours, ncl_strlen_0) ;
>>>>> initial_time0:NCL_converted_from_type = "string" ;
>>>>> initial_time0:units = "mm/dd/yyyy (hh:mm)" ;
>>>>> initial_time0:long_name = "Initial time of first
>>>>> record" ;
>>>>> int forecast_time1(forecast_time1) ;
>>>>> forecast_time1:units = "hours" ;
>>>>> forecast_time1:long_name = "Forecast offset from
>>>>> initial time" ;
>>>>> float g4_lon_3(g4_lon_3) ;
>>>>> g4_lon_3:La1 = 89.463f ;
>>>>> g4_lon_3:Lo1 = 0.f ;
>>>>> g4_lon_3:La2 = -89.463f ;
>>>>> g4_lon_3:Lo2 = 359.297f ;
>>>>> g4_lon_3:Di = 0.7031252f ;
>>>>> g4_lon_3:N = 128.f ;
>>>>> g4_lon_3:units = "degrees_east" ;
>>>>> g4_lon_3:GridType = "Gaussian Latitude/Longitude Grid"
>>>>> ;
>>>>> g4_lon_3:long_name = "longitude" ;
>>>>> float g4_lat_2(g4_lat_2) ;
>>>>> g4_lat_2:La1 = 89.463f ;
>>>>> g4_lat_2:Lo1 = 0.f ;
>>>>> g4_lat_2:La2 = -89.463f ;
>>>>> g4_lat_2:Lo2 = 359.297f ;
>>>>> g4_lat_2:Di = 0.7031252f ;
>>>>> g4_lat_2:N = 128.f ;
>>>>> g4_lat_2:units = "degrees_north" ;
>>>>> g4_lat_2:GridType = "Gaussian Latitude/Longitude Grid"
>>>>> ;
>>>>> g4_lat_2:long_name = "latitude" ;
>>>>> double initial_time0_encoded(initial_time0_hours) ;
>>>>> initial_time0_encoded:units = "yyyymmddhh.hh_frac" ;
>>>>> initial_time0_encoded:long_name = "initial time encoded
>>>>>
>>>>> as double" ;
>>>>> double initial_time0_hours(initial_time0_hours) ;
>>>>> initial_time0_hours:units = "hours since 1800-01-01
>>>>> 00:00" ;
>>>>> initial_time0_hours:long_name = "initial time" ;
>>>>> float STRD_GDS4_SFC(initial_time0_hours, forecast_time1,
>>>>> g4_lat_2, g4_lon_3) ;
>>>>> STRD_GDS4_SFC:parameter_number = 175 ;
>>>>> STRD_GDS4_SFC:parameter_table_version = 128 ;
>>>>> STRD_GDS4_SFC:gds_grid_type = 4 ;
>>>>> STRD_GDS4_SFC:level_indicator = 1 ;
>>>>> STRD_GDS4_SFC:_FillValue = 1.e+20f ;
>>>>> STRD_GDS4_SFC:units = "W m**-2 s" ;
>>>>> STRD_GDS4_SFC:long_name = "Surface thermal radiation
>>>>> downwards" ;
>>>>> STRD_GDS4_SFC:center = "European Center for
>>>>> Medium-Range Weather Forecasts (RSMC)" ;
>>>>>
>>>>> // global attributes:
>>>>> :creation_date = "Thu Apr 26 15:49:03 MDT 2012" ;
>>>>> :NCL_Version = "6.0.0" ;
>>>>> :system = "AIX be1005en 3 5 00CBF2314C00 AIX" ;
>>>>> :conventions = "None" ;
>>>>> :grib_source = "input.grb" ;
>>>>> :title = "NCL: convert-GRIB-to-netCDF" ;
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ncl-talk mailing list
>>>>> List instructions, subscriber options, unsubscribe:
>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>> _______________________________________________
>>>> ncl-talk mailing list
>>>> List instructions, subscriber options, unsubscribe:
>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>> _______________________________________________
>>> ncl-talk mailing list
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>
>> --
>> Associate Research Professor of Climatology
>> Dept of Earth and Environmental Science
>> MSEC 304; 801 Leroy Place
>> New Mexico Tech
>> Socorro, NM 87801
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>

-- 
Associate Research Professor of Climatology
Dept of Earth and Environmental Science
MSEC 304; 801 Leroy Place
New Mexico Tech
Socorro, NM 87801

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Mon May 7 17:42:09 2012

This archive was generated by hypermail 2.1.8 : Thu May 10 2012 - 16:57:50 MDT