Re: Compressed netCDF with NCL 6.0.0-beta

From: Mary Haley <haley_at_nyahnyahspammersnyahnyah>
Date: Thu Mar 10 2011 - 21:42:38 MST

The problem was with an older gcc compiler that I was using.

I rebuilt the Mac binary with a new compiler, and uploaded a new
"ncl_ncarg-6.0.0-beta.MacOS_10.6_64bit_gcc421.tar.gz" file, if anybody
wants to download it again.

If you don't use NetCDF compression, then you probably don't need to
get this new file.

--Mary

On Mar 10, 2011, at 8:12 AM, Mary Haley wrote:

> Hi Robert,
>
> I'm looking into this. Robert, I think the "non-OpeNDAP" binary
> should work fine
> with compression. There's something strange about this particular
> NCL. I couldn't
> reproduce this problem on Linux systems, so I'm hoping it's limited
> to the Mac
> for the OPeNDAP-enabled binary only.
>
> --Mary
>
> On Mar 10, 2011, at 7:07 AM, Dennis Shea wrote:
>
>> The test code ran successfully on all systems under v5.2.1
>> Under v6.0.0-beta, it ran successfully on all systems *except* a MAC.
>> It did not get the same "warning" problem(s) as you encountered.
>> However, it did encounter a "segmentation fault" even when writing
>> nc3.
>>
>> Please provide the output from:
>>
>> %> uname -a
>> %> gcc --version
>>
>> THX
>> D
>> On 3/10/11 1:26 AM, Robert Schuster wrote:
>>> Hello Dave and Dennis,
>>>
>>> the problem is not that compression is too slow or that it creates
>>> an
>>> overhead on small files. The problem is that I'm not able to use
>>> compression at all. Your test script is, like mine, leading to a
>>> segmentation fault:
>>>
>>> Copyright (C) 1995-2011 - All Rights Reserved
>>> University Corporation for Atmospheric Research
>>> NCAR Command Language Version 6.0.0-beta
>>> The use of this software is governed by a License Agreement.
>>> See http://www.ncl.ucar.edu/ for more details.
>>> warning:stringtointeger: a bad value was passed; input strings must
>>> contain numeric digits, replacing with missing value
>>> warning:stringtointeger: a bad value was passed; input strings must
>>> contain numeric digits, replacing with missing value
>>> (0)
>>> =====> Wall Clock Elapsed Time: test_nc3.1000.nc: -2147483647
>>> seconds <=====
>>>
>>> warning:stringtointeger: a bad value was passed; input strings must
>>> contain numeric digits, replacing with missing value
>>> warning:stringtointeger: a bad value was passed; input strings must
>>> contain numeric digits, replacing with missing value
>>> (0)
>>> =====> Wall Clock Elapsed Time: test_nc3.1000000.nc: -2147483647
>>> seconds
>>> <=====
>>>
>>> warning:stringtointeger: a bad value was passed; input strings must
>>> contain numeric digits, replacing with missing value
>>> warning:stringtointeger: a bad value was passed; input strings must
>>> contain numeric digits, replacing with missing value
>>> (0)
>>> =====> Wall Clock Elapsed Time: test_nc3.10000000.nc: -2147483647
>>> seconds <=====
>>>
>>> warning:stringtointeger: a bad value was passed; input strings must
>>> contain numeric digits, replacing with missing value
>>> warning:stringtointeger: a bad value was passed; input strings must
>>> contain numeric digits, replacing with missing value
>>> (0)
>>> =====> Wall Clock Elapsed Time: test_nc3.100000000.nc: -2147483647
>>> seconds <=====
>>>
>>> Segmentation fault
>>>
>>> The segmentation fault is caused by the setfileoption procedure:
>>> Copyright (C) 1995-2011 - All Rights Reserved
>>> University Corporation for Atmospheric Research
>>> NCAR Command Language Version 6.0.0-beta
>>> The use of this software is governed by a License Agreement.
>>> See http://www.ncl.ucar.edu/ for more details.
>>> ncl 0> setfileoption("nc", "Format", "NetCDF4Classic")
>>> Segmentation fault
>>>
>>> Copyright (C) 1995-2011 - All Rights Reserved
>>> University Corporation for Atmospheric Research
>>> NCAR Command Language Version 6.0.0-beta
>>> The use of this software is governed by a License Agreement.
>>> See http://www.ncl.ucar.edu/ for more details.
>>> ncl 0> setfileoption("nc", "CompressionLevel", 5)
>>> Segmentation fault
>>>
>>> I don't have that problem with NCL 5.2.0 on my machine.
>>>
>>> Thank you for your help!
>>> Robert
>>>
>>>
>>> Am 09.03.2011 um 22:07 schrieb Dennis Shea:
>>>
>>>> Dave and I communicated offline. There is overhead when
>>>> creating a compressed file. I have attached a simple test script.
>>>> Run on v6.0.0 beta
>>>>
>>>> Bottom line:
>>>> (a) Use compression for large files only
>>>> (b) Compression can be slow for larger files.
>>>> Using compression levels > 1 is probably
>>>> not worth the extra time needed to create the file.
>>>> Even compression level 1 is slow.
>>>> --------------------------------------------------------
>>>> netCDF3
>>>> 4080 Mar 9 13:45 test_nc3.1000.nc
>>>> 4000080 Mar 9 13:45 test_nc3.1000000.nc
>>>> 40000080 Mar 9 13:45 test_nc3.10000000.nc
>>>> 400000080 Mar 9 13:45 test_nc3.100000000.nc
>>>>
>>>> netCDF4: no compression
>>>>
>>>> 11215 Mar 9 13:35 test_nc4.cl0.1000.nc <note size overhead
>>>> 4007820 Mar 9 13:35 test_nc4.cl0.1000000.nc
>>>> 41956704 Mar 9 13:35 test_nc4.cl0.10000000.nc
>>>> 402726596 Mar 9 13:35 test_nc4.cl0.100000000.nc
>>>>
>>>> netCDF4: compression level 1
>>>>
>>>> 10805 Mar 9 13:45 test_nc4.cl1.1000.nc
>>>> 3398074 Mar 9 13:45 test_nc4.cl1.1000000.nc
>>>> 33923350 Mar 9 13:45 test_nc4.cl1.10000000.nc
>>>> 339070469 Mar 9 13:46 test_nc4.cl1.100000000.nc
>>>>
>>>> netCDF4: compression level 5
>>>>
>>>> 10789 Mar 9 13:33 test_nc4.cl5.1000.nc
>>>> 3365173 Mar 9 13:33 test_nc4.cl5.1000000.nc
>>>> 33597009 Mar 9 13:33 test_nc4.cl5.10000000.nc
>>>> 335844543 Mar 9 13:34 test_nc4.cl5.100000000.nc
>>>>
>>>> Elapsed time to create each file:
>>>>
>>>> =====> Wall Clock Elapsed Time: test_nc3.1000.nc: 0 seconds
>>>> =====> Wall Clock Elapsed Time: test_nc3.1000000.nc: 0 seconds
>>>> =====> Wall Clock Elapsed Time: test_nc3.10000000.nc: 0 seconds
>>>> =====> Wall Clock Elapsed Time: test_nc3.100000000.nc: 3 seconds
>>>>
>>>> =====> Wall Clock Elapsed Time: test_nc4.cl1.1000.nc: 0 seconds
>>>> =====> Wall Clock Elapsed Time: test_nc4.cl1.1000000.nc: 1 seconds
>>>> =====> Wall Clock Elapsed Time: test_nc4.cl1.10000000.nc: 3 seconds
>>>> =====> Wall Clock Elapsed Time: test_nc4.cl1.100000000.nc: 31
>>>> seconds **
>>>>
>>>> =====> Wall Clock Elapsed Time: test_nc4.cl5.1000.nc: 0 seconds
>>>> =====> Wall Clock Elapsed Time: test_nc4.cl5.1000000.nc: 0 seconds
>>>> =====> Wall Clock Elapsed Time: test_nc4.cl5.10000000.nc: 4 seconds
>>>> =====> Wall Clock Elapsed Time: test_nc4.cl5.100000000.nc: 43
>>>> seconds **
>>>>
>>>>
>>>>
>>>>
>>>> On 03/09/2011 11:14 AM, David Brown wrote:
>>>>> I cannot reproduce this error with ncl 6.0.0-beta either on mac
>>>>> 10.6.x
>>>>> or on an x86_64 Linux machine. Can you let us know more details
>>>>> about
>>>>> your environment? Which binary? Built from source, etc?
>>>>> -dave
>>>>>
>>>>> On Mar 9, 2011, at 10:59 AM, Robert Schuster wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> is it not possible to create compressed netCDF-Files with NCL
>>>>>> 6.0.0-beta?
>>>>>>
>>>>>> This small script is leading to a segmentation fault:
>>>>>>
>>>>>> begin
>>>>>> setfileoption("nc", "Format", "NetCDF4Classic")
>>>>>> setfileoption("nc", "CompressionLevel", 5)
>>>>>>
>>>>>> var = new (1000,float)
>>>>>> outfile = addfile("test.nc","c")
>>>>>> outfile->var = var
>>>>>> delete(outfile)
>>>>>> end
>>>>>>
>>>>>> Thank you!
>>>>>> Robert
>>>>>>
>>>>>> _______________________________________________
>>>>>> ncl-talk mailing list
>>>>>> List instructions, subscriber options, unsubscribe:
>>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ncl-talk mailing list
>>>>> List instructions, subscriber options, unsubscribe:
>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>
>>>>
>>>> --
>>>> ======================================================
>>>> Dennis J. Shea tel: 303-497-1361 |
>>>> P.O. Box 3000 fax: 303-497-1333 |
>>>> Climate Analysis Section |
>>>> Climate & Global Dynamics Div. |
>>>> National Center for Atmospheric Research |
>>>> Boulder, CO 80307 |
>>>> USA email: shea 'at' ucar.edu <http://ucar.edu> |
>>>> ======================================================
>>>> <compress.ncl>
>>>
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Thu Mar 10 21:42:45 2011

This archive was generated by hypermail 2.1.8 : Thu Mar 10 2011 - 21:44:37 MST