Re: compression levels for ncl_convert2nc

Hi Jonathan,

You probably are aware that NetCDF compression is only available for NetCDF4, which uses the HDF5 file format under the covers. NetCDF4 compression uses zlib, a library with a loss-less compression algorithm no matter what the compression level. The only difference between compression levels is the amount of time it takes to perform compression and decompression, and the eventual size of the data. In practice, little additional compression is gained beyond level 2 (or maybe even 1, I forget right now). The higher levels do take considerably more time however.

By default, NetCDF data is not compressed at all. On the other hand, the primary virtue of GRIB is its compaction of the data, some of which may be achieved by limiting the precision of the data to less than the full precision of a single-precision IEEE float type.

So to answer your questions directly: (1) the NetCDF4 compression levels have *no* effect on the precision of the data and (2) no it does not follow. GRIB files generally will always be smaller, even compared to compressed NetCDF4 files. 3 times the size sounds about right for an uncompressed NetCDF file compared to a GRIB file with the same data.

You should not lose precision by converting GRIB to NetCDF at any compression level.
 -dave

On Jun 15, 2012, at 4:05 PM, Jonathan Vigh wrote:

> Greetings NCL'ers,
>
> I'm contemplating the use of ncl_convert2nc to convert some large model output files from grib format to netcdf. These netcdf files will be used later to construct a climatology of the model data set over many months, so I need to pay some attention to the precision in order to compute accurate anomalies. I've done some tests -- if I use the default settings on ncl_convert2nc (equivalent to a compression level of 0), the resulting file is nearly three times the size of the original grib file. I'd like to preserve as much practical numerical fidelity as possible without adding in "wasteful" precision that doesn't add any information. So my questions are:
> Does anyone have any information about how the different compression levels affect precision?
> Does it follow that the precision in a grib file or netcdf file will be similar if their file sizes are similar?
> I have no idea if data stored in a netcdf vs. grib should be of similar sizes. I thought they might be, since they are both a type of binary, except I realize that the details of how the data are packed could lead to significant differences.
>
> Thanks,
> Jonathan
>
> --
> Jonathan Vigh
> Project Scientist I, Joint Numerical Testbed
> Research Applications Laboratory (RAL)
> National Center for Atmospheric Research (NCAR)
> P.O. Box 3000 tel: +1 303 497 8205
> Boulder, CO 80307-3000 fax: +1 (303) 497-8171
> http://www.ral.ucar.edu/staff/jvigh/
> http://www.ral.ucar.edu/hurricanes/
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Fri Jun 15 17:04:40 2012