Re: Shuffle filter for Netcdf-4 compression

Dave B,

Okay, I did not understand that shuffle was enabled for non-default.
I had not gotten that far in my testing. I'm happy with this
behavior, so allow me to withdraw my request. So far in my very
limited testing, shuffle only improves the compression ratio.

I think the shuffle filter should be mentioned in the documentation,
probably within the paragraph on CompressionLevel. The shuffle
setting becomes visible to users with ncdump -s, so I think it would
be good to warn of how this works.

Another thing I just found the hard way is that
setfileoption(CompressionLevel) must be called *before* filevardef.
Please mention the required calling order for CompressionLevel. I
suppose it also works this way for direct var assignment, e.g. f->x =
x.

Let me know if I'm still mixed up about any of this. Thank you, your
explanation was very helpful.

--Dave A.

On Thu, Dec 20, 2012 at 6:21 PM, David Brown <dbrown@ucar.edu> wrote:
> Hi Dave,
> The original code for classic mode NetCDF4 turns the shuffle filter on whenever the CompressionLevel is set to a non-default value. Since then code has been modified to allow for compression on a per-variable basis and I am not sure if that statement is still true in every case. Since we now have finer-grained control of compression, it may make sense to allow for specific control of the shuffle filter. I am not sure if there are cases where turning the shuffle filter off would improve compression ratio or performance, so I don't know if the option would be useful. My original reading of the docs (and in the spirit of trying to keep it simple) led me to decide to just turn the shuffle filter on unconditionally.
> -dave
>
> On Dec 20, 2012, at 5:12 PM, Dave Allured - NOAA Affiliate wrote:
>
>> NCL team,
>>
>> Could you please add a setfileoption to enable the HDF5 built-in
>> shuffle filter? As explained in HDF5 docs, this is a companion to
>> other compression filters. It seems like the code for this on/off
>> filter setting should be simple. In the Netcdf C API, this is simply
>> a separate argument in the deflate function call, so you don't need
>> any fancy access into the HDF layer. Thanks.
>>
>> http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-c.html#nc_005fdef_005fvar_005fdeflate
>>
>> HDF5: "When the shuffle filter is applied to a dataset immediately
>> prior to the use of a compression filter, the compression ratio
>> achieved is often superior to that achieved by the use of a
>> compression filter without the shuffle filter."
>>
>> http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetShuffle
>>
>> --Dave
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Thu Dec 20 19:24:10 2012