Re: Compressed netCDF with NCL 6.0.0-beta

Hi Robert,

I'm looking into this. Robert, I think the "non-OpeNDAP" binary should work fine
with compression. There's something strange about this particular NCL. I couldn't
reproduce this problem on Linux systems, so I'm hoping it's limited to the Mac
for the OPeNDAP-enabled binary only.

--Mary

On Mar 10, 2011, at 7:07 AM, Dennis Shea wrote:

> The test code ran successfully on all systems under v5.2.1
> Under v6.0.0-beta, it ran successfully on all systems *except* a MAC.
> It did not get the same "warning" problem(s) as you encountered.
> However, it did encounter a "segmentation fault" even when writing nc3.
>
> Please provide the output from:
>
> %> uname -a
> %> gcc --version
>
> THX
> D
> On 3/10/11 1:26 AM, Robert Schuster wrote:
>> Hello Dave and Dennis,
>>
>> the problem is not that compression is too slow or that it creates an
>> overhead on small files. The problem is that I'm not able to use
>> compression at all. Your test script is, like mine, leading to a
>> segmentation fault:
>>
>> Copyright (C) 1995-2011 - All Rights Reserved
>> University Corporation for Atmospheric Research
>> NCAR Command Language Version 6.0.0-beta
>> The use of this software is governed by a License Agreement.
>> See http://www.ncl.ucar.edu/ for more details.
>> warning:stringtointeger: a bad value was passed; input strings must
>> contain numeric digits, replacing with missing value
>> warning:stringtointeger: a bad value was passed; input strings must
>> contain numeric digits, replacing with missing value
>> (0)
>> =====> Wall Clock Elapsed Time: test_nc3.1000.nc: -2147483647 seconds <=====
>>
>> warning:stringtointeger: a bad value was passed; input strings must
>> contain numeric digits, replacing with missing value
>> warning:stringtointeger: a bad value was passed; input strings must
>> contain numeric digits, replacing with missing value
>> (0)
>> =====> Wall Clock Elapsed Time: test_nc3.1000000.nc: -2147483647 seconds
>> <=====
>>
>> warning:stringtointeger: a bad value was passed; input strings must
>> contain numeric digits, replacing with missing value
>> warning:stringtointeger: a bad value was passed; input strings must
>> contain numeric digits, replacing with missing value
>> (0)
>> =====> Wall Clock Elapsed Time: test_nc3.10000000.nc: -2147483647
>> seconds <=====
>>
>> warning:stringtointeger: a bad value was passed; input strings must
>> contain numeric digits, replacing with missing value
>> warning:stringtointeger: a bad value was passed; input strings must
>> contain numeric digits, replacing with missing value
>> (0)
>> =====> Wall Clock Elapsed Time: test_nc3.100000000.nc: -2147483647
>> seconds <=====
>>
>> Segmentation fault
>>
>> The segmentation fault is caused by the setfileoption procedure:
>> Copyright (C) 1995-2011 - All Rights Reserved
>> University Corporation for Atmospheric Research
>> NCAR Command Language Version 6.0.0-beta
>> The use of this software is governed by a License Agreement.
>> See http://www.ncl.ucar.edu/ for more details.
>> ncl 0> setfileoption("nc", "Format", "NetCDF4Classic")
>> Segmentation fault
>>
>> Copyright (C) 1995-2011 - All Rights Reserved
>> University Corporation for Atmospheric Research
>> NCAR Command Language Version 6.0.0-beta
>> The use of this software is governed by a License Agreement.
>> See http://www.ncl.ucar.edu/ for more details.
>> ncl 0> setfileoption("nc", "CompressionLevel", 5)
>> Segmentation fault
>>
>> I don't have that problem with NCL 5.2.0 on my machine.
>>
>> Thank you for your help!
>> Robert
>>
>>
>> Am 09.03.2011 um 22:07 schrieb Dennis Shea:
>>
>>> Dave and I communicated offline. There is overhead when
>>> creating a compressed file. I have attached a simple test script.
>>> Run on v6.0.0 beta
>>>
>>> Bottom line:
>>> (a) Use compression for large files only
>>> (b) Compression can be slow for larger files.
>>> Using compression levels > 1 is probably
>>> not worth the extra time needed to create the file.
>>> Even compression level 1 is slow.
>>> --------------------------------------------------------
>>> netCDF3
>>> 4080 Mar 9 13:45 test_nc3.1000.nc
>>> 4000080 Mar 9 13:45 test_nc3.1000000.nc
>>> 40000080 Mar 9 13:45 test_nc3.10000000.nc
>>> 400000080 Mar 9 13:45 test_nc3.100000000.nc
>>>
>>> netCDF4: no compression
>>>
>>> 11215 Mar 9 13:35 test_nc4.cl0.1000.nc <note size overhead
>>> 4007820 Mar 9 13:35 test_nc4.cl0.1000000.nc
>>> 41956704 Mar 9 13:35 test_nc4.cl0.10000000.nc
>>> 402726596 Mar 9 13:35 test_nc4.cl0.100000000.nc
>>>
>>> netCDF4: compression level 1
>>>
>>> 10805 Mar 9 13:45 test_nc4.cl1.1000.nc
>>> 3398074 Mar 9 13:45 test_nc4.cl1.1000000.nc
>>> 33923350 Mar 9 13:45 test_nc4.cl1.10000000.nc
>>> 339070469 Mar 9 13:46 test_nc4.cl1.100000000.nc
>>>
>>> netCDF4: compression level 5
>>>
>>> 10789 Mar 9 13:33 test_nc4.cl5.1000.nc
>>> 3365173 Mar 9 13:33 test_nc4.cl5.1000000.nc
>>> 33597009 Mar 9 13:33 test_nc4.cl5.10000000.nc
>>> 335844543 Mar 9 13:34 test_nc4.cl5.100000000.nc
>>>
>>> Elapsed time to create each file:
>>>
>>> =====> Wall Clock Elapsed Time: test_nc3.1000.nc: 0 seconds
>>> =====> Wall Clock Elapsed Time: test_nc3.1000000.nc: 0 seconds
>>> =====> Wall Clock Elapsed Time: test_nc3.10000000.nc: 0 seconds
>>> =====> Wall Clock Elapsed Time: test_nc3.100000000.nc: 3 seconds
>>>
>>> =====> Wall Clock Elapsed Time: test_nc4.cl1.1000.nc: 0 seconds
>>> =====> Wall Clock Elapsed Time: test_nc4.cl1.1000000.nc: 1 seconds
>>> =====> Wall Clock Elapsed Time: test_nc4.cl1.10000000.nc: 3 seconds
>>> =====> Wall Clock Elapsed Time: test_nc4.cl1.100000000.nc: 31 seconds **
>>>
>>> =====> Wall Clock Elapsed Time: test_nc4.cl5.1000.nc: 0 seconds
>>> =====> Wall Clock Elapsed Time: test_nc4.cl5.1000000.nc: 0 seconds
>>> =====> Wall Clock Elapsed Time: test_nc4.cl5.10000000.nc: 4 seconds
>>> =====> Wall Clock Elapsed Time: test_nc4.cl5.100000000.nc: 43 seconds **
>>>
>>>
>>>
>>>
>>> On 03/09/2011 11:14 AM, David Brown wrote:
>>>> I cannot reproduce this error with ncl 6.0.0-beta either on mac 10.6.x
>>>> or on an x86_64 Linux machine. Can you let us know more details about
>>>> your environment? Which binary? Built from source, etc?
>>>> -dave
>>>>
>>>> On Mar 9, 2011, at 10:59 AM, Robert Schuster wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> is it not possible to create compressed netCDF-Files with NCL
>>>>> 6.0.0-beta?
>>>>>
>>>>> This small script is leading to a segmentation fault:
>>>>>
>>>>> begin
>>>>> setfileoption("nc", "Format", "NetCDF4Classic")
>>>>> setfileoption("nc", "CompressionLevel", 5)
>>>>>
>>>>> var = new (1000,float)
>>>>> outfile = addfile("test.nc","c")
>>>>> outfile->var = var
>>>>> delete(outfile)
>>>>> end
>>>>>
>>>>> Thank you!
>>>>> Robert
>>>>>
>>>>> _______________________________________________
>>>>> ncl-talk mailing list
>>>>> List instructions, subscriber options, unsubscribe:
>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> ncl-talk mailing list
>>>> List instructions, subscriber options, unsubscribe:
>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>>
>>> --
>>> ======================================================
>>> Dennis J. Shea tel: 303-497-1361 |
>>> P.O. Box 3000 fax: 303-497-1333 |
>>> Climate Analysis Section |
>>> Climate & Global Dynamics Div. |
>>> National Center for Atmospheric Research |
>>> Boulder, CO 80307 |
>>> USA email: shea 'at' ucar.edu <http://ucar.edu> |
>>> ======================================================
>>> <compress.ncl>
>>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Thu Mar 10 08:12:42 2011