Re: Shuffle filter for Netcdf-4 compression

From: David Brown <dbrown_at_nyahnyahspammersnyahnyah>
Date: Thu Dec 20 2012 - 18:21:00 MST

Hi Dave,
The original code for classic mode NetCDF4 turns the shuffle filter on whenever the CompressionLevel is set to a non-default value. Since then code has been modified to allow for compression on a per-variable basis and I am not sure if that statement is still true in every case. Since we now have finer-grained control of compression, it may make sense to allow for specific control of the shuffle filter. I am not sure if there are cases where turning the shuffle filter off would improve compression ratio or performance, so I don't know if the option would be useful. My original reading of the docs (and in the spirit of trying to keep it simple) led me to decide to just turn the shuffle filter on unconditionally.
 -dave

On Dec 20, 2012, at 5:12 PM, Dave Allured - NOAA Affiliate wrote:

> NCL team,
>
> Could you please add a setfileoption to enable the HDF5 built-in
> shuffle filter? As explained in HDF5 docs, this is a companion to
> other compression filters. It seems like the code for this on/off
> filter setting should be simple. In the Netcdf C API, this is simply
> a separate argument in the deflate function call, so you don't need
> any fancy access into the HDF layer. Thanks.
>
> http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-c.html#nc_005fdef_005fvar_005fdeflate
>
> HDF5: "When the shuffle filter is applied to a dataset immediately
> prior to the use of a compression filter, the compression ratio
> achieved is often superior to that achieved by the use of a
> compression filter without the shuffle filter."
>
> http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetShuffle
>
> --Dave
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Thu Dec 20 18:21:14 2012

This archive was generated by hypermail 2.1.8 : Fri Dec 21 2012 - 10:43:23 MST