Re: compression levels for ncl_convert2nc

Hi Dave,

This is extremely helpful. Thanks!

A follow-up question: does it take longer to read a nc4 file that was
compressed than one that was written with no compression?
If so, speed may be more important than size for me.

Thanks again,
    Jonathan

On 06/15/2012 11:04 PM, David Brown wrote:
> Hi Jonathan,
>
> You probably are aware that NetCDF compression is only available for
> NetCDF4, which uses the HDF5 file format under the covers. NetCDF4
> compression uses zlib, a library with a loss-less compression
> algorithm no matter what the compression level. The only difference
> between compression levels is the amount of time it takes to perform
> compression and decompression, and the eventual size of the data. In
> practice, little additional compression is gained beyond level 2 (or
> maybe even 1, I forget right now). The higher levels do take
> considerably more time however.
>
> By default, NetCDF data is not compressed at all. On the other hand,
> the primary virtue of GRIB is its compaction of the data, some of
> which may be achieved by limiting the precision of the data to less
> than the full precision of a single-precision IEEE float type.
>
> So to answer your questions directly: (1) the NetCDF4 compression
> levels have *no* effect on the precision of the data and (2) no it
> does not follow. GRIB files generally will always be smaller, even
> compared to compressed NetCDF4 files. 3 times the size sounds about
> right for an uncompressed NetCDF file compared to a GRIB file with the
> same data.
>
> You should not lose precision by converting GRIB to NetCDF at any
> compression level.
> -dave
>
> On Jun 15, 2012, at 4:05 PM, Jonathan Vigh wrote:
>
>> Greetings NCL'ers,
>>
>> I'm contemplating the use of ncl_convert2nc to convert some large
>> model output files from grib format to netcdf. These netcdf files
>> will be used later to construct a climatology of the model data set
>> over many months, so I need to pay some attention to the precision in
>> order to compute accurate anomalies. I've done some tests -- if I use
>> the default settings on ncl_convert2nc (equivalent to a compression
>> level of 0), the resulting file is nearly three times the size of the
>> original grib file. I'd like to preserve as much practical numerical
>> fidelity as possible without adding in "wasteful" precision that
>> doesn't add any information. So my questions are:
>>
>> * Does anyone have any information about how the different
>> compression levels affect precision?
>> * Does it follow that the precision in a grib file or netcdf file
>> will be similar if their file sizes are similar?
>>
>> I have no idea if data stored in a netcdf vs. grib should be of
>> similar sizes. I thought they might be, since they are both a type of
>> binary, except I realize that the details of how the data are packed
>> could lead to significant differences.
>>
>> Thanks,
>> Jonathan
>>
>> --
>> Jonathan Vigh
>> Project Scientist I, Joint Numerical Testbed
>> Research Applications Laboratory (RAL)
>> National Center for Atmospheric Research (NCAR)
>> P.O. Box 3000 tel: +1 303 497 8205
>> Boulder, CO 80307-3000 fax: +1 (303) 497-8171
>> http://www.ral.ucar.edu/staff/jvigh/
>> http://www.ral.ucar.edu/hurricanes/
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>

-- 
Jonathan Vigh						
Project Scientist I, Joint Numerical Testbed	
Research Applications Laboratory (RAL)
National Center for Atmospheric Research (NCAR)
P.O. Box 3000            tel: +1 303 497 8205
Boulder, CO 80307-3000   fax: +1 (303) 497-8171
http://www.ral.ucar.edu/staff/jvigh/
http://www.ral.ucar.edu/hurricanes/

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Fri Jun 15 17:11:24 2012