Re: NCL data compression (Ravi Shekhar)

Ravi,

You can enable compression in netCDF. Depending on how up-to-date your
netCDF library is, you can use the "nccopy" command
(http://www.unidata.ucar.edu/software/netcdf/docs/netcdf/nccopy.html).
This utility will read a given netCDF file and convert it to netCDF4 and
turn on HDF5 compression. You might want to experiment with the
"deflation" level parameter. The higher it goes, the more compression,
but the marginal increase in compression diminishes. Also, reading
compressed files into netCDF will take a longer because the data has to
be uncompressed on the fly.

A word about NCO: When concatenating, make sure the time variable in all
the files uses the same reference time ("days since ..."). If they don't
NCO will take the first reference time as the overall reference time.

Regards,

Chad

On 10/21/2012 02:00 PM, ncl-talk-request@ucar.edu wrote:
> Send ncl-talk mailing list submissions to
> ncl-talk@ucar.edu
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
> or, via email, send a message with subject or body 'help' to
> ncl-talk-request@ucar.edu
>
> You can reach the person managing the list at
> ncl-talk-owner@ucar.edu
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of ncl-talk digest..."
>
>
> Today's Topics:
>
> 1. NCL data compression (Ravi Shekhar)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sun, 21 Oct 2012 11:53:38 -0400
> From: Ravi Shekhar <ravi.shekhar@yale.edu>
> Subject: NCL data compression
> To: ncl-talk@ucar.edu
> Message-ID: <8794126A623249ECAF1944895E5DF524@yale.edu>
> Content-Type: text/plain; charset="utf-8"
>
> Hi,
>
> I have some WRF output that is very disk intensive (pushing into the several terabyte range). It is output in NetCDF 64 bit offset file format.
>
> Is it possible for me to enable some sort of compression on this data? Perhaps after converting to NetCDF-4, which, as I understand it, supports compression? I did a quick test with gzip, and `gzip -1` reduces size to about 1/3 of previous, which would be extremely helpful.
>
> All of my analysis code is in NCL, so conversion to any file format NCL supports should be fine. I have to concatenate a large number of files anyway, so I was going to use `ncrcat -4` from the NCO utilities to do the conversion.
>
> Thanks,
> Ravi
>

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Sun Oct 21 12:41:48 2012