Re: Compressed netCDF with NCL 6.0.0-beta

From: Dennis Shea <shea_at_nyahnyahspammersnyahnyah>
Date: Wed Mar 09 2011 - 14:07:04 MST

Dave and I communicated offline. There is overhead when
creating a compressed file. I have attached a simple test script.
Run on v6.0.0 beta

Bottom line:
   (a) Use compression for large files only
   (b) Compression can be slow for larger files.
       Using compression levels > 1 is probably
       not worth the extra time needed to create the file.
       Even compression level 1 is slow.
--------------------------------------------------------
netCDF3
        4080 Mar 9 13:45 test_nc3.1000.nc
     4000080 Mar 9 13:45 test_nc3.1000000.nc
    40000080 Mar 9 13:45 test_nc3.10000000.nc
   400000080 Mar 9 13:45 test_nc3.100000000.nc

netCDF4: no compression

       11215 Mar 9 13:35 test_nc4.cl0.1000.nc <note size overhead
     4007820 Mar 9 13:35 test_nc4.cl0.1000000.nc
    41956704 Mar 9 13:35 test_nc4.cl0.10000000.nc
   402726596 Mar 9 13:35 test_nc4.cl0.100000000.nc

netCDF4: compression level 1

       10805 Mar 9 13:45 test_nc4.cl1.1000.nc
     3398074 Mar 9 13:45 test_nc4.cl1.1000000.nc
    33923350 Mar 9 13:45 test_nc4.cl1.10000000.nc
   339070469 Mar 9 13:46 test_nc4.cl1.100000000.nc

netCDF4: compression level 5

       10789 Mar 9 13:33 test_nc4.cl5.1000.nc
     3365173 Mar 9 13:33 test_nc4.cl5.1000000.nc
    33597009 Mar 9 13:33 test_nc4.cl5.10000000.nc
   335844543 Mar 9 13:34 test_nc4.cl5.100000000.nc

Elapsed time to create each file:

=====> Wall Clock Elapsed Time: test_nc3.1000.nc: 0 seconds
=====> Wall Clock Elapsed Time: test_nc3.1000000.nc: 0 seconds
=====> Wall Clock Elapsed Time: test_nc3.10000000.nc: 0 seconds
=====> Wall Clock Elapsed Time: test_nc3.100000000.nc: 3 seconds

=====> Wall Clock Elapsed Time: test_nc4.cl1.1000.nc: 0 seconds
=====> Wall Clock Elapsed Time: test_nc4.cl1.1000000.nc: 1 seconds
=====> Wall Clock Elapsed Time: test_nc4.cl1.10000000.nc: 3 seconds
=====> Wall Clock Elapsed Time: test_nc4.cl1.100000000.nc: 31 seconds **

=====> Wall Clock Elapsed Time: test_nc4.cl5.1000.nc: 0 seconds
=====> Wall Clock Elapsed Time: test_nc4.cl5.1000000.nc: 0 seconds
=====> Wall Clock Elapsed Time: test_nc4.cl5.10000000.nc: 4 seconds
=====> Wall Clock Elapsed Time: test_nc4.cl5.100000000.nc: 43 seconds **

On 03/09/2011 11:14 AM, David Brown wrote:
> I cannot reproduce this error with ncl 6.0.0-beta either on mac 10.6.x
> or on an x86_64 Linux machine. Can you let us know more details about
> your environment? Which binary? Built from source, etc?
> -dave
>
> On Mar 9, 2011, at 10:59 AM, Robert Schuster wrote:
>
>> Hello,
>>
>> is it not possible to create compressed netCDF-Files with NCL 6.0.0-beta?
>>
>> This small script is leading to a segmentation fault:
>>
>> begin
>> setfileoption("nc", "Format", "NetCDF4Classic")
>> setfileoption("nc", "CompressionLevel", 5)
>>
>> var = new (1000,float)
>> outfile = addfile("test.nc","c")
>> outfile->var = var
>> delete(outfile)
>> end
>>
>> Thank you!
>> Robert
>>
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

-- 
======================================================
Dennis J. Shea                  tel: 303-497-1361    |
P.O. Box 3000                   fax: 303-497-1333    |
Climate Analysis Section                             |
Climate & Global Dynamics Div.                       |
National Center for Atmospheric Research             |
Boulder, CO  80307                                   |
USA                        email: shea 'at' ucar.edu |
======================================================


_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk

Received on Wed Mar 9 14:07:11 2011

This archive was generated by hypermail 2.1.8 : Thu Mar 10 2011 - 21:44:37 MST