ncl-talk 2011 archive: Re: Compressed netCDF with NCL 6.0.0-beta

From: Robert Schuster <rxschuster_at_nyahnyahspammersnyahnyah>
Date: Thu Mar 10 2011 - 01:26:50 MST

Hello Dave and Dennis,

the problem is not that compression is too slow or that it creates an overhead on small files. The problem is that I'm not able to use compression at all. Your test script is, like mine, leading to a segmentation fault:

Copyright (C) 1995-2011 - All Rights Reserved
University Corporation for Atmospheric Research
NCAR Command Language Version 6.0.0-beta
The use of this software is governed by a License Agreement.
See http://www.ncl.ucar.edu/ for more details.
warning:stringtointeger: a bad value was passed; input strings must contain numeric digits, replacing with missing value
warning:stringtointeger: a bad value was passed; input strings must contain numeric digits, replacing with missing value
(0)
=====> Wall Clock Elapsed Time: test_nc3.1000.nc: -2147483647 seconds <=====

warning:stringtointeger: a bad value was passed; input strings must contain numeric digits, replacing with missing value
warning:stringtointeger: a bad value was passed; input strings must contain numeric digits, replacing with missing value
(0)
=====> Wall Clock Elapsed Time: test_nc3.1000000.nc: -2147483647 seconds <=====

Segmentation fault

The segmentation fault is caused by the setfileoption procedure:
Copyright (C) 1995-2011 - All Rights Reserved
University Corporation for Atmospheric Research
NCAR Command Language Version 6.0.0-beta
The use of this software is governed by a License Agreement.
See http://www.ncl.ucar.edu/ for more details.
ncl 0> setfileoption("nc", "Format", "NetCDF4Classic")
Segmentation fault

Copyright (C) 1995-2011 - All Rights Reserved
University Corporation for Atmospheric Research
NCAR Command Language Version 6.0.0-beta
The use of this software is governed by a License Agreement.
See http://www.ncl.ucar.edu/ for more details.
ncl 0> setfileoption("nc", "CompressionLevel", 5)
Segmentation fault

I don't have that problem with NCL 5.2.0 on my machine.

Thank you for your help!
Robert

Am 09.03.2011 um 22:07 schrieb Dennis Shea:

> Dave and I communicated offline. There is overhead when
> creating a compressed file. I have attached a simple test script.
> Run on v6.0.0 beta
>
> Bottom line:
> (a) Use compression for large files only
> (b) Compression can be slow for larger files.
> Using compression levels > 1 is probably
> not worth the extra time needed to create the file.
> Even compression level 1 is slow.
> --------------------------------------------------------
> netCDF3
> 4080 Mar 9 13:45 test_nc3.1000.nc
> 4000080 Mar 9 13:45 test_nc3.1000000.nc
> 40000080 Mar 9 13:45 test_nc3.10000000.nc
> 400000080 Mar 9 13:45 test_nc3.100000000.nc
>
> netCDF4: no compression
>
> 11215 Mar 9 13:35 test_nc4.cl0.1000.nc <note size overhead
> 4007820 Mar 9 13:35 test_nc4.cl0.1000000.nc
> 41956704 Mar 9 13:35 test_nc4.cl0.10000000.nc
> 402726596 Mar 9 13:35 test_nc4.cl0.100000000.nc
>
> netCDF4: compression level 1
>
> 10805 Mar 9 13:45 test_nc4.cl1.1000.nc
> 3398074 Mar 9 13:45 test_nc4.cl1.1000000.nc
> 33923350 Mar 9 13:45 test_nc4.cl1.10000000.nc
> 339070469 Mar 9 13:46 test_nc4.cl1.100000000.nc
>
> netCDF4: compression level 5
>
> 10789 Mar 9 13:33 test_nc4.cl5.1000.nc
> 3365173 Mar 9 13:33 test_nc4.cl5.1000000.nc
> 33597009 Mar 9 13:33 test_nc4.cl5.10000000.nc
> 335844543 Mar 9 13:34 test_nc4.cl5.100000000.nc
>
> Elapsed time to create each file:
>
> =====> Wall Clock Elapsed Time: test_nc3.1000.nc: 0 seconds
> =====> Wall Clock Elapsed Time: test_nc3.1000000.nc: 0 seconds
> =====> Wall Clock Elapsed Time: test_nc3.10000000.nc: 0 seconds
> =====> Wall Clock Elapsed Time: test_nc3.100000000.nc: 3 seconds
>
> =====> Wall Clock Elapsed Time: test_nc4.cl1.1000.nc: 0 seconds
> =====> Wall Clock Elapsed Time: test_nc4.cl1.1000000.nc: 1 seconds
> =====> Wall Clock Elapsed Time: test_nc4.cl1.10000000.nc: 3 seconds
> =====> Wall Clock Elapsed Time: test_nc4.cl1.100000000.nc: 31 seconds **
>
> =====> Wall Clock Elapsed Time: test_nc4.cl5.1000.nc: 0 seconds
> =====> Wall Clock Elapsed Time: test_nc4.cl5.1000000.nc: 0 seconds
> =====> Wall Clock Elapsed Time: test_nc4.cl5.10000000.nc: 4 seconds
> =====> Wall Clock Elapsed Time: test_nc4.cl5.100000000.nc: 43 seconds **
>
>
>
>
> On 03/09/2011 11:14 AM, David Brown wrote:
>> I cannot reproduce this error with ncl 6.0.0-beta either on mac 10.6.x
>> or on an x86_64 Linux machine. Can you let us know more details about
>> your environment? Which binary? Built from source, etc?
>> -dave
>>
>> On Mar 9, 2011, at 10:59 AM, Robert Schuster wrote:
>>
>>> Hello,
>>>
>>> is it not possible to create compressed netCDF-Files with NCL 6.0.0-beta?
>>>
>>> This small script is leading to a segmentation fault:
>>>
>>> begin
>>> setfileoption("nc", "Format", "NetCDF4Classic")
>>> setfileoption("nc", "CompressionLevel", 5)
>>>
>>> var = new (1000,float)
>>> outfile = addfile("test.nc","c")
>>> outfile->var = var
>>> delete(outfile)
>>> end
>>>
>>> Thank you!
>>> Robert
>>>
>>> _______________________________________________
>>> ncl-talk mailing list
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>>
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
> --
> ======================================================
> Dennis J. Shea tel: 303-497-1361 |
> P.O. Box 3000 fax: 303-497-1333 |
> Climate Analysis Section |
> Climate & Global Dynamics Div. |
> National Center for Atmospheric Research |
> Boulder, CO 80307 |
> USA email: shea 'at' ucar.edu |
> ======================================================
> <compress.ncl>

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Thu Mar 10 01:27:06 2011

This archive was generated by hypermail 2.1.8 : Thu Mar 10 2011 - 21:44:37 MST