ncl_convert2nc

ncl_convert2nc converts one or more GRIB1, GRIB2, HDF 4, HDF-EOS 2, HDF-EOS 5, netCDF, and/or shapefile files to netCDF formatted files.

Synopsis

  ncl_convert2nc inputFile(s) OPTIONS
    inputFile(s)          name(s) of data file(s) [required] 
                          [valid types: GRIB1 GRIB2 HDF HDF-EOS netCDF shapefile]
    [-i input_directory]  location of input file(s)  [default: current directory]
    [-o output_directory] location of output file(s) [default: current directory]
    [-e extension]        file type, defined by extension, to convert [example: grb]
    [-u time_name]        name of the NCL-named time dimension to be UNLIMITED
    [-U new_time_name]    if -u is specified: new name of UNLIMITED variable and dimension
    [-sed sed1[,...]]     GRIB files only; set single element dimensions [default: none]
                            choices are initial_time, forecast_time, level, ensemble,
                            probability, all, none
    [-itime]              GRIB files only; set initial time as a single element dimension
                            (same as -sed initial_time)
    [-ftime]              GRIB files only; set forecast time as a single element dimension
                            (same as -sed forecast_time)
    [-tps]                GRIB files only; remove suffix representing a time period (e.g. 2h)
                            from statistically processed variables, leaving only type of
                            processing as a suffix (e.g. _acc, _avg)
    [-v var1[,...]]       user specified subset of variables [default: all variables]
                          ncl_filedump can be used to determine desired variable names
    [-L]                  support for writing large (>2Gb) netCDF files [default: no largefile support]
    [-nc4]                output a netCDF 4 file
    [-nc4c]               output a netCDF 4 "classic" format file 
    [-cl compression_level]  compression level 0 - 9 [ default 0 - only meaningful for netCDF 4 output]
    [-th size_in_megabytes]  theshold size: variables larger than specified size will be written incrementally
                          in order to limit in-core memory requirements; 0 (the default) means no threshold
    [-c comment]          text to be included in netCDF file attribute [default: no comment]
    [-l]                  list some information about each variable as it is copied
    [-d]                  upon exit: print contents of each netCDF file [like ncdump -h]
    [-B] <file>           suppress informational messages; redirect messages to <file> if present
                          [default: /dev/null]
    [-h]                  this usage message

Description

ncl_convert2nc converts one or more GRIB1 and GRIB2, HDF 4, HDF-EOS 2, or HDF-EOS 5 files to netCDF formatted files. Each input file must be a GRIB, HDF, HDF-EOS, HDF-EOS5, shapefile, or netCDF file that NCL can read. The output filenames will have the same name as the input filenames, except the output filenames will have the '.nc' extension ('.nc4' if the output file is NetCDF 4). Each valid input file will result in the creation of a corresponding netCDF output file.

As of version 5.2.0, ncl_convert2nc can function as a converter between different netCDF formats as well as to select a subset of the variables in a netCDF file or change the compression level of netCDF 4 classic files.

Support was added for HDF-EOS5 files in version 5.2.0.

Beta test support was added for shapefile in version 5.1.1. The NCL binaries for AIX, Cygwin, and Sun systems may not have shapefile support available.

Support was added for GRIB2 files in version 4.3.0.

Options

ncl_convert2nc accepts the following arguments and options:

inputFile(s)

The name of the input GRIB, HDF, HDF-EOS and/or HDF-EOS5 file(s). More than one input filename may be specified; in this case, each file is processed individually. Specify a file extension, e.g. .grb to indicate the type of data contained in the input file. For many files of the same type, use the -e option. This argument is required.

-i input_directory

The location of input files. Defaults to the current working directory.

-o output_directory

The location where output files will be written. Defaults to the current working directory.

-e extension

Specifies the file extension to be applied to input file(s) that do not have an extension denoting their type. The value for extension must be one of the valid extensions for data types GRIB, HDF, HDF-EOS, or HDF-EOS5.

-u time_name

The name of the NCL-named time dimension to be dimensioned as UNLIMITED.

-U new_time_name

If '-u' is specified: new name of the NCL-named time dimension to be dimensioned as UNLIMITED.

-sed sed1[,...]

Sets the single element record dimensions for GRIB files; choices are Initial_time, Forecast_time, Level, Ensemble, Probability, All, and None. The default is None.

-itime

Set time record dimension to initial_time. This allows a variable with only a single initial time to be treated as a single element dimension.

-ftime

Set time record dimension to initial_time. This allows a variable with only a single forecast time to be treated as a single element dimension.

-tps

If specified, removes the suffix representing a time period (e.g. 2h) from statistically processed variables, leaving only the type of processing as a suffix (e.g. "_acc", "_avg", etc.) for GRIB files. The default is True, which leaves the suffix in place.

-v var1[,...]

Specifies one or more variable names to be included in the output netCDF file. One or more variables must be specified by name, in a comma-delimited list (without blanks or other whitespace characters). Named variables must be valid within the input file.

-L

Specifies that the resultant netCDF output file may exceed 2Gb in size on platforms that have "large file support" (LFS). However, no single variable may exceed 2Gb in the current implementation.

-nc4

Specifies the netCDF 4 format for the output file.
[NCL V6.5.0 and later]

-nc4c

Specifies the netCDF 4 "classic" format for the output file.

-cl compression_level

Sets the compression level for all variables in a netCDF 4 file. The valid range is 0 - 9, with 0 meaning no compression and 9 meaning maximum compression. In practice, values above 5 produce little addition compression but take much more time to complete.

-th size_in_megabytes

If greater than 0, sets a threshold size for variables being copied. Variables larger than the threshold size will be copied in chunks to limit the in-core memory requirements needed for the copying operation. On machines without a lot of memory this option often can significantly shorten the time required. However, if writing a compressed netCDF 4 file, it can have the opposite effect because the the compression algorithm may be operating across the chunk boundaries.

-c comment

User-supplied text to be included as a netCDF file attribute called "comment."

-l

List some information about each variable as it is processed. This can be used to guage the progress of the conversion operation when large files are processed.

-d

Print the contents of each output netCDF file before exiting the process.

-B <file>

Suppresses informational messages emitted by ncl_convert2nc. If >file< is provided, messages are redirected there. Defaults to /dev/null (messages discarded).

-h

Display a short message explaining the command usage, and exit.

Examples

Example 1

The following converts the input GRIB file 'U12345.grb' to a netCDF-formatted file called 'U12345.nc' written in the current working directory. The resultant netCDF file is shown (in part).

          % ncl_convert2nc U12345.grb
          [...]
          filename:       U12345
          path:   U12345.nc
             file global attributes:
                creation_date : Wed Jan  2 13:47:15 MST 2018
                NCL_Version : 6.5.0
                conventions : None
                grib_source : U12345.grb
                title : NCL: convert-GRIB-to-netCDF

Example 2

The following converts the input GRIB files 'U12345' and 'U67890' to netCDF-formatted files called 'U12345.nc' and 'U67890.nc' written in the current working directory. It also creates a singly dimensioned element for initial time. The resultant netCDF files are shown (in part).

          % ncl_convert2nc U12345 U67890 -e grb -itime
          [...]
          filename:       U12345
          path:   U12345.nc
             file global attributes:
                creation_date : Wed Nov  2 13:47:15 MST 2017
                NCL_Version : 6.4.0
                conventions : None
                grib_source : U12345
                title : NCL: convert-GRIB-to-netCDF

          filename:       U67890
          path:   U67890.nc
             file global attributes:
                creation_date : Wed Nov  2 13:48:21 MST 2017
                NCL_Version : 6.4.0
                conventions : None
                grib_source : U67890
                title : NCL: convert-GRIB-to-netCDF
          dimensions:
             initial_time0_hours = 1

Example 3

The following converts the input GRIB file 'U12345' to a netCDF-formatted file called 'U12345.nc' written in the current working directory. Note that specifying the input file as 'U12345.grb' tells ncl_convert2nc to treat the file as GRIB data, even though there is no extension to indicate this, nor is the -e extension option specified. The resultant netCDF file is shown (in part).

          % ncl_convert2nc U12345.grb
          [...]
          filename:       U12345
          path:   U12345.nc
             file global attributes:
                creation_date : Wed Nov  2 13:47:15 MST 2017
                NCL_Version : 6.4.0
                conventions : None
                grib_source : U12345
                title : NCL: convert-GRIB-to-netCDF

Example 4

The following converts the input GRIB file 'U12345.grb' and HDF file 'U67890.hdf' located in the directory /data/dss/INPUT to netCDF-formatted files called 'U12345.nc' and 'U67890.nc' written in the directory /data/dss/netCDF. The resultant netCDF files are shown (in part).

          % ncl_convert2nc U12345.grb U67890.hdf -i /data/dss/INPUT -o /data/dss/netCDF
          [...]
          filename:       U12345
          path:   /data/dss/netCDF/U12345.nc
             file global attributes:
                creation_date : Wed Nov  2 13:47:15 MST 2017
                NCL_Version : 6.4.0
                conventions : None
                grib_source : U12345.grb
                title : NCL: convert-GRIB-to-netCDF

          [...] 

          filename:       U67890
          path:   /data/dss/netCDF/U67890.nc
             file global attributes:
                creation_date : Wed Nov  2 13:48:12 MST 2017
                NCL_Version : 6.4.0
                conventions : None
                hdf_source : U67890.hdf
                title : NCL: convert-HDF-to-netCDF

Example 5

The following converts the input GRIB file 'U12345.grb' and HDF-EOS file '/data/hdfeos/05062005.hdfeos' to netCDF-formatted files called 'U12345.nc' and '05062005.nc' written in the current working directory. The resultant netCDF files are shown(in part).

          % ncl_convert2nc U12345.grb /data/hdfeos/05062005.hdfeos
          [...]
          filename:       U12345
          path:   U12345.nc
             file global attributes:
                creation_date : Wed Nov  2 13:47:15 MST 2017
                NCL_Version : 6.4.0
                conventions : None
                grib_source : U12345.grb
                title : NCL: convert-GRIB-to-netCDF

          [...] 

          filename:       05062005
          path:   05062005.nc
             file global attributes:
                creation_date : Wed Nov  2 13:48:15 MST 2017
                NCL_Version : 6.4.0
                conventions : None
                hdfeos_source : /data/hdfeos/05062005.hdfeos
                title : NCL: convert-HDFEOS-to-netCDF

Example 6

The following converts the input HDF file 'U12345.hdf' located in the current working directory to a netCDF-formatted file called 'U12345.nc' written in the current working directory and sets the netCDF 'comment' attribute to the text 'Data Support Section: ds124.1'. Any informational and/or warning messages are suppressed. The resultant netCDF file is shown (in part).

          % ncl_convert2nc U12345.hdf -c 'Data Support Section: ds124.1' -B

          [...]

          filename:       U12345
          path:   U12345.nc
             file global attributes:
                comment: Data Support Section: ds124.1
                creation_date : Wed Nov  2 13:47:15 MST 2017
                NCL_Version : 6.4.0
                conventions : None
                hdf_source : U12345.hdf
                title : NCL: convert-HDF-to-netCDF

Example 7

The following converts the input GRIB file 'U12345.grb' located in the current working directory to a netCDF-formatted file called 'U12345.nc' written in the current working directory and writes only the variables 'PRES_6_SFC' and 'PRES_6_TRO' to the output file. The output file supports netCDF-formatted "large files;" that is, files larger than 2Gb in size. The resultant netCDF file is shown (in part).

          % ncl_convert2nc U12345.grb -v PRES_6_SFC,PRES_6_TRO -L
          [...]
          filename:       U12345
          path:   U12345.nc
             file global attributes:
                creation_date : Wed Nov  2 13:47:15 MST 2017
                NCL_Version : 6.4.0
                conventions : None
                grib_source : U12345.grb
                title : NCL: convert-GRIB-to-netCDF

          [...]

          variables:
             float PRES_6_SFC ( gridx_6, gridy_6 )
                initial_time : 10/24/1995 (00:00)
                forecast_time :        0
                parameter_number :     1
                grid_number :  6
                level_indicator :      1
                coordinates :  gridlat_6 gridlon_6
                units :        Pa
                long_name :    Pressure
                center :       US National Weather Service - NCEP (WMC)
                _FillValue :   -999

             float PRES_6_TRO ( gridx_6, gridy_6 )
                initial_time : 10/24/1995 (00:00)
                forecast_time :        0
                parameter_number :     1
                grid_number :  6
                level_indicator :      7
                coordinates :  gridlat_6 gridlon_6
                units :        Pa
                long_name :    Pressure
                center :       US National Weather Service - NCEP (WMC)
                _FillValue :   -999

Example 8

The following prints the usage message:

           % ncl_convert2nc -h
             ncl_convert2nc inputFile(s) OPTIONS
               inputFile(s)          name(s) of data file(s) [required] 
                                     [valid types: GRIB1 GRIB2 HDF HDF-EOS netCDF shapefile]
               [-i input_directory]  location of input file(s)  [default: current directory]
               [-o output_directory] location of output file(s) [default: current directory]
               [-e extension]        file type, defined by extension, to convert [example: grb]
               [-u time_name]        name of the NCL-named time dimension to be UNLIMITED
               [-U new_time_name]    if -u is specified: new name of UNLIMITED variable and dimension
               [-sed sed1[,...]]     GRIB files only; set single element dimensions [default: none]
                                       choices are initial_time, forecast_time, level, ensemble,
                                       probability, all, none
               [-itime]              GRIB files only; set initial time as a single element dimension
                                       (same as -sed initial_time)
               [-ftime]              GRIB files only; set forecast time as a single element dimension
                                       (same as -sed forecast_time)
               [-tps]                GRIB files only; remove suffix representing a time period (e.g. 2h)
                                       from statistically processed variables, leaving only type of
                                       processing as a suffix (e.g. _acc, _avg)
               [-v var1[,...]]       user specified subset of variables [default: all variables]
                                     ncl_filedump can be used to determine desired variable names
               [-L]                  support for writing large (>2Gb) netCDF files [default: no largefile support]
               [-nc4]                output a netCDF 4 file
               [-nc4c]               output a NetCDF 4 classic format file
               [-cl]   compression level 0 - 9 [ default 0 - only meaningful for NetCDF 4 output]
               [-th]  theshold size: variables larger than specified size will be written incrementally
                                     in order to limit in-core memory requirements; 0 (the default) means no threshold
               [-c comment]          text to be included in netCDF file attribute [default: no comment]
               [-l]                  list some information about each variable as it is copied
               [-d]                  upon exit: print contents of each netCDF file [like ncdump -h]
               [-B] <file>           suppress informational messages; redirect messages to <file> if present 
                                     [default: /dev/null]
               [-h]                  this usage message

Troubleshooting

If you try to convert a file to netCDF and get the following error message:

   ncendef: ncid 0: NetCDF: One or more variable sizes violate format
   constraints

Then you may have what qualifies as a "large file", and you need to use the "-L" option with ncl_convertnc.

Caveats

To convert Community Climate Model (CCM) files, please use the ccm2nc tool, available at the CCM Software Tools site.

It is important to note is that GRIB is a read-only format -- GRIB files can not be created using NCL. This is because, in general, the community that NCL serves does not use GRIB as an archival format.

There is currently limited capability within NCL to write files in HDF format.

There is currently no capability within NCL to write files in HDF-EOS and HDF-EOS5 format.

NCL does not currently support HDF 5 or netCDF 4 (except NetCDF 4 classic model) data formats.

Files written using the netCDF-supported large file format may be greater than 2Gb in size; however, at this time NCL does not support writing single variables greater than 2Gb.