Re: Compressed netCDF with NCL 6.0.0-beta

The test code ran successfully on all systems under v5.2.1
Under v6.0.0-beta, it ran successfully on all systems *except* a MAC.
It did not get the same "warning" problem(s) as you encountered.
However, it did encounter a "segmentation fault" even when writing nc3.

Please provide the output from:

%> uname -a
%> gcc --version

THX
D
On 3/10/11 1:26 AM, Robert Schuster wrote:
> Hello Dave and Dennis,
>
> the problem is not that compression is too slow or that it creates an
> overhead on small files. The problem is that I'm not able to use
> compression at all. Your test script is, like mine, leading to a
> segmentation fault:
>
> Copyright (C) 1995-2011 - All Rights Reserved
> University Corporation for Atmospheric Research
> NCAR Command Language Version 6.0.0-beta
> The use of this software is governed by a License Agreement.
> See http://www.ncl.ucar.edu/ for more details.
> warning:stringtointeger: a bad value was passed; input strings must
> contain numeric digits, replacing with missing value
> warning:stringtointeger: a bad value was passed; input strings must
> contain numeric digits, replacing with missing value
> (0)
> =====> Wall Clock Elapsed Time: test_nc3.1000.nc: -2147483647 seconds <=====
>
> warning:stringtointeger: a bad value was passed; input strings must
> contain numeric digits, replacing with missing value
> warning:stringtointeger: a bad value was passed; input strings must
> contain numeric digits, replacing with missing value
> (0)
> =====> Wall Clock Elapsed Time: test_nc3.1000000.nc: -2147483647 seconds
> <=====
>
> warning:stringtointeger: a bad value was passed; input strings must
> contain numeric digits, replacing with missing value
> warning:stringtointeger: a bad value was passed; input strings must
> contain numeric digits, replacing with missing value
> (0)
> =====> Wall Clock Elapsed Time: test_nc3.10000000.nc: -2147483647
> seconds <=====
>
> warning:stringtointeger: a bad value was passed; input strings must
> contain numeric digits, replacing with missing value
> warning:stringtointeger: a bad value was passed; input strings must
> contain numeric digits, replacing with missing value
> (0)
> =====> Wall Clock Elapsed Time: test_nc3.100000000.nc: -2147483647
> seconds <=====
>
> Segmentation fault
>
> The segmentation fault is caused by the setfileoption procedure:
> Copyright (C) 1995-2011 - All Rights Reserved
> University Corporation for Atmospheric Research
> NCAR Command Language Version 6.0.0-beta
> The use of this software is governed by a License Agreement.
> See http://www.ncl.ucar.edu/ for more details.
> ncl 0> setfileoption("nc", "Format", "NetCDF4Classic")
> Segmentation fault
>
> Copyright (C) 1995-2011 - All Rights Reserved
> University Corporation for Atmospheric Research
> NCAR Command Language Version 6.0.0-beta
> The use of this software is governed by a License Agreement.
> See http://www.ncl.ucar.edu/ for more details.
> ncl 0> setfileoption("nc", "CompressionLevel", 5)
> Segmentation fault
>
> I don't have that problem with NCL 5.2.0 on my machine.
>
> Thank you for your help!
> Robert
>
>
> Am 09.03.2011 um 22:07 schrieb Dennis Shea:
>
>> Dave and I communicated offline. There is overhead when
>> creating a compressed file. I have attached a simple test script.
>> Run on v6.0.0 beta
>>
>> Bottom line:
>> (a) Use compression for large files only
>> (b) Compression can be slow for larger files.
>> Using compression levels > 1 is probably
>> not worth the extra time needed to create the file.
>> Even compression level 1 is slow.
>> --------------------------------------------------------
>> netCDF3
>> 4080 Mar 9 13:45 test_nc3.1000.nc
>> 4000080 Mar 9 13:45 test_nc3.1000000.nc
>> 40000080 Mar 9 13:45 test_nc3.10000000.nc
>> 400000080 Mar 9 13:45 test_nc3.100000000.nc
>>
>> netCDF4: no compression
>>
>> 11215 Mar 9 13:35 test_nc4.cl0.1000.nc <note size overhead
>> 4007820 Mar 9 13:35 test_nc4.cl0.1000000.nc
>> 41956704 Mar 9 13:35 test_nc4.cl0.10000000.nc
>> 402726596 Mar 9 13:35 test_nc4.cl0.100000000.nc
>>
>> netCDF4: compression level 1
>>
>> 10805 Mar 9 13:45 test_nc4.cl1.1000.nc
>> 3398074 Mar 9 13:45 test_nc4.cl1.1000000.nc
>> 33923350 Mar 9 13:45 test_nc4.cl1.10000000.nc
>> 339070469 Mar 9 13:46 test_nc4.cl1.100000000.nc
>>
>> netCDF4: compression level 5
>>
>> 10789 Mar 9 13:33 test_nc4.cl5.1000.nc
>> 3365173 Mar 9 13:33 test_nc4.cl5.1000000.nc
>> 33597009 Mar 9 13:33 test_nc4.cl5.10000000.nc
>> 335844543 Mar 9 13:34 test_nc4.cl5.100000000.nc
>>
>> Elapsed time to create each file:
>>
>> =====> Wall Clock Elapsed Time: test_nc3.1000.nc: 0 seconds
>> =====> Wall Clock Elapsed Time: test_nc3.1000000.nc: 0 seconds
>> =====> Wall Clock Elapsed Time: test_nc3.10000000.nc: 0 seconds
>> =====> Wall Clock Elapsed Time: test_nc3.100000000.nc: 3 seconds
>>
>> =====> Wall Clock Elapsed Time: test_nc4.cl1.1000.nc: 0 seconds
>> =====> Wall Clock Elapsed Time: test_nc4.cl1.1000000.nc: 1 seconds
>> =====> Wall Clock Elapsed Time: test_nc4.cl1.10000000.nc: 3 seconds
>> =====> Wall Clock Elapsed Time: test_nc4.cl1.100000000.nc: 31 seconds **
>>
>> =====> Wall Clock Elapsed Time: test_nc4.cl5.1000.nc: 0 seconds
>> =====> Wall Clock Elapsed Time: test_nc4.cl5.1000000.nc: 0 seconds
>> =====> Wall Clock Elapsed Time: test_nc4.cl5.10000000.nc: 4 seconds
>> =====> Wall Clock Elapsed Time: test_nc4.cl5.100000000.nc: 43 seconds **
>>
>>
>>
>>
>> On 03/09/2011 11:14 AM, David Brown wrote:
>>> I cannot reproduce this error with ncl 6.0.0-beta either on mac 10.6.x
>>> or on an x86_64 Linux machine. Can you let us know more details about
>>> your environment? Which binary? Built from source, etc?
>>> -dave
>>>
>>> On Mar 9, 2011, at 10:59 AM, Robert Schuster wrote:
>>>
>>>> Hello,
>>>>
>>>> is it not possible to create compressed netCDF-Files with NCL
>>>> 6.0.0-beta?
>>>>
>>>> This small script is leading to a segmentation fault:
>>>>
>>>> begin
>>>> setfileoption("nc", "Format", "NetCDF4Classic")
>>>> setfileoption("nc", "CompressionLevel", 5)
>>>>
>>>> var = new (1000,float)
>>>> outfile = addfile("test.nc","c")
>>>> outfile->var = var
>>>> delete(outfile)
>>>> end
>>>>
>>>> Thank you!
>>>> Robert
>>>>
>>>> _______________________________________________
>>>> ncl-talk mailing list
>>>> List instructions, subscriber options, unsubscribe:
>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>>
>>>
>>> _______________________________________________
>>> ncl-talk mailing list
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>> --
>> ======================================================
>> Dennis J. Shea tel: 303-497-1361 |
>> P.O. Box 3000 fax: 303-497-1333 |
>> Climate Analysis Section |
>> Climate & Global Dynamics Div. |
>> National Center for Atmospheric Research |
>> Boulder, CO 80307 |
>> USA email: shea 'at' ucar.edu <http://ucar.edu> |
>> ======================================================
>> <compress.ncl>
>
_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Thu Mar 10 07:07:21 2011