Re: Error when writing global attributes in Netcdf-4 classic file

From: Wei Huang <huangwei_at_nyahnyahspammersnyahnyah>
Date: Tue Apr 02 2013 - 13:28:07 MDT

For tracking purpose, I have created a JIRA ticket 1692.

Wei

huangwei@ucar.edu
VETS/CISL
National Center for Atmospheric Research
P.O. Box 3000 (1850 Table Mesa Dr.)
Boulder, CO 80307-3000 USA
(303) 497-8924

On Apr 2, 2013, at 12:04 PM, Dave Allured - NOAA Affiliate <dave.allured@noaa.gov> wrote:

> Wei,
>
> Thanks for checking this out. There is definitely a problem
> somewhere. Now I am finding the same seg fault on Linux systems as
> well as different Mac OS machines, with both NCL 6.0.0 and 6.1.2. In
> fact I can not find a local system that does NOT get this seg fault.
>
> Might this be related to supplemental libraries? Following are simple
> dependency lists for both Linux and Mac. Let me know if there are any
> other diagnostics that I can provide. This is not a high priority for
> me, I have a workaround.
>
> Dave
>
> inta:~/temp/3-atts/402a 1045> date
> Tue Apr 2 11:49:25 MDT 2013
>
> inta:~/temp/3-atts/402a 1046> uname -a
> Linux inta.psd.esrl.noaa.gov 2.6.18-348.3.1.el5 #1 SMP Tue Mar 5
> 13:19:32 EST 2013 x86_64 GNU/Linux
>
> inta:~/temp/3-atts/402a 1047> echo $NCARG_ROOT
> /usr/local/ncl6.1.2
>
> inta:~/temp/3-atts/402a 1048> ls -go $NCARG_ROOT/bin/ncl
> -rwxr-xr-x 1 32917555 Feb 6 21:30 /usr/local/ncl6.1.2/bin/ncl
>
> inta:~/temp/3-atts/402a 1049> ls -go
> -rw-r--r-- 1 501 Apr 2 10:55 demo.wei-huang.0401a.ncl
> -r--r--r-- 1 11294147 Apr 1 17:14 ozone.master.nc
>
> inta:~/temp/3-atts/402a 1050> cat demo.wei-huang.0401a.ncl
> begin
> outfile = "out.nc"
> ; system ("cp ozone.master.nc " + outfile) ; create fresh writable copy
> system ("cat ozone.master.nc > " + outfile) ; create fresh writable copy
> system ("chmod +w " + outfile) ; create fresh writable copy
>
> out = addfile (outfile, "w")
>
> out@last_update = systemfunc ("date")
> out@last_update_program = "demo.global-atts.ncl version 2013-apr-01"
> out@last_update_source_file = "ei.ozone.1979010106.nc"
> print ("Update complete.")
> end
>
> inta:~/temp/3-atts/402a 1051> $NCARG_ROOT/bin/ncl demo.wei-huang.0401a.ncl
> Copyright (C) 1995-2013 - All Rights Reserved
> University Corporation for Atmospheric Research
> NCAR Command Language Version 6.1.2
> The use of this software is governed by a License Agreement.
> See http://www.ncl.ucar.edu/ for more details.
> (0) Update complete.
> Segmentation fault
>
> inta:~/temp/3-atts/402a 1052> ldd $NCARG_ROOT/bin/ncl
> linux-vdso.so.1 => (0x00007fff7dbfd000)
> libcurl.so.3 => /usr/lib64/libcurl.so.3 (0x0000003288400000)
> libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003278600000)
> libbz2.so.1 => /usr/lib64/libbz2.so.1 (0x000000328c000000)
> libXrender.so.1 => /usr/lib64/libXrender.so.1 (0x000000327aa00000)
> libX11.so.6 => /usr/lib64/libX11.so.6 (0x0000003278e00000)
> libXext.so.6 => /usr/lib64/libXext.so.6 (0x0000003279a00000)
> libm.so.6 => /lib64/libm.so.6 (0x0000003277a00000)
> libdl.so.2 => /lib64/libdl.so.2 (0x0000003277e00000)
> libstdc++.so.6 => /usr/local/gcc-4.7/lib/libstdc++.so.6 (0x00002b9f6bc6c000)
> libgcc_s.so.1 => /usr/local/gcc-4.7/lib/libgcc_s.so.1 (0x00002b9f6bf76000)
> libc.so.6 => /lib64/libc.so.6 (0x0000003277600000)
> /lib64/ld-linux-x86-64.so.2 (0x0000003277200000)
> libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x0000003282c00000)
> libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x0000003282400000)
> libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x0000003282000000)
> libcom_err.so.2 => /lib64/libcom_err.so.2 (0x0000003281c00000)
> libidn.so.11 => /usr/lib64/libidn.so.11 (0x0000003284000000)
> libssl.so.6 => /lib64/libssl.so.6 (0x0000003288c00000)
> libcrypto.so.6 => /lib64/libcrypto.so.6 (0x0000003288800000)
> libz.so.1 => /lib64/libz.so.1 (0x0000003278200000)
> libXau.so.6 => /usr/lib64/libXau.so.6 (0x0000003279600000)
> libXdmcp.so.6 => /usr/lib64/libXdmcp.so.6 (0x0000003278a00000)
> libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x0000003282800000)
> libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x0000003281400000)
> libresolv.so.2 => /lib64/libresolv.so.2 (0x0000003280c00000)
> libselinux.so.1 => /lib64/libselinux.so.1 (0x000000327d200000)
> libsepol.so.1 => /lib64/libsepol.so.1 (0x000000327ce00000)
>
> ========================================================
>
> mac56:~/ncl/bugs/3-atts/402a 35> uname -a
> Darwin mac56 12.3.0 Darwin Kernel Version 12.3.0: Sun Jan 6 22:37:10
> PST 2013; root:xnu-2050.22.13~1/RELEASE_X86_64 x86_64
>
> mac56:~/ncl/bugs/3-atts/402a 36> echo $NCARG_ROOT
> /usr/local/ncl6.1.2
>
> mac56:~/ncl/bugs/3-atts/402a 37> ls -dgo $NCARG_ROOT/bin/ncl
> -rwxr-xr-x 1 33112624 Feb 6 21:38 /usr/local/ncl6.1.2/bin/ncl
>
> mac56:~/ncl/bugs/3-atts/402a 38> ldd $NCARG_ROOT/bin/ncl
> /usr/local/ncl6.1.2/bin/ncl:
> /usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 7.0.0)
> /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
> /usr/X11/lib/libXrender.1.dylib (compatibility version 5.0.0, current
> version 5.0.0)
> /usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.5)
> /usr/X11/lib/libXpm.4.dylib (compatibility version 16.0.0, current
> version 16.0.0)
> /usr/X11/lib/libX11.6.dylib (compatibility version 10.0.0, current
> version 10.0.0)
> /usr/X11/lib/libXext.6.dylib (compatibility version 11.0.0, current
> version 11.0.0)
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 159.1.0)
> /usr/local/lib/libstdc++.6.dylib (compatibility version 7.0.0,
> current version 7.14.0)
> /usr/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current
> version 1.0.0)
>
> On Mon, Apr 1, 2013 at 9:35 PM, Wei Huang <huangwei@ucar.edu> wrote:
>> Dave,
>>
>> I can not reproduce your problem.
>>
>> I only added:
>> system ("chmod +w " + outfile)
>> after system call in your script (as the original file is write protected).
>>
>> and it runs fine.
>>
>> I tested on Mac, with osx 10.8.3.
>>
>> Here is the full script I tested:
>>
>> ; 2013-apr-01 Test program reduced from era-update.ncl.
>> ; Demonstrates seg fault when trying to overwrite global
>> ; attributes in a Netcdf-4 classic file.
>> ; By Dave Allured, NOAA/PSD/CIRES Climate Analysis Branch.
>>
>> begin
>> outfile = "out.nc"
>> ; system ("cp ozone.master.nc " + outfile) ; create fresh writable copy
>> system ("cat ozone.master.nc > " + outfile) ; create fresh writable copy
>> system ("chmod +w " + outfile) ; create fresh writable copy
>>
>> out = addfile (outfile, "w")
>>
>> out@last_update = systemfunc ("date")
>> out@last_update_program = "demo.global-atts.ncl version 2013-apr-01"
>> out@last_update_source_file = "ei.ozone.1979010106.nc"
>> print ("Update complete.")
>> end
>>
>>
>> and part of ncdump -h out.nc:
>>
>> // global attributes:
>> :title = "ERA-Interim Ozone mass mixing ratio, 4 times daily" ;
>> :history = "Mon Apr 1 17:14:49 MDT 2013:\n",
>> "Initial file made by era-create.ncl version 2013-apr-01 TEST" ;
>> :dataset = "ERA-Interim" ;
>> :version = "2.0" ;
>> :references = "http://www.ecmwf.int/research/era/do/get/era-interim;\n",
>> "D. P. Dee et al (2011), The ERA-Interim reanalysis:\n",
>> " configuration and performance of the data assimilation system,\n",
>> " Q.J.R. Meteorol. Soc., Vol. 137: 553-597, DOI: 10.1002/qj.828" ;
>> :center = "European Center for Medium-Range Weather Forecasts (RSMC)" ;
>> :metadata_source_file = "../ei.oper.an.pl.regn128sc.1979010100" ;
>> :Conventions = "CF-1.6" ;
>> :last_update = "Mon Apr 1 21:28:20 MDT 2013" ;
>> :last_update_program = "demo.global-atts.ncl version 2013-apr-01" ;
>> :last_update_source_file = "ei.ozone.1979010106.nc" ;
>> }
>>
>>
>> Wei
>>
>> huangwei@ucar.edu
>> VETS/CISL
>> National Center for Atmospheric Research
>> P.O. Box 3000 (1850 Table Mesa Dr.)
>> Boulder, CO 80307-3000 USA
>> (303) 497-8924
>>
>>
>>
>>
>>
>>
>> On Apr 1, 2013, at 8:41 PM, Dave Allured - NOAA Affiliate <dave.allured@noaa.gov> wrote:
>>
>>> NCL support team,
>>>
>>> This program results in a seg fault error on program exit, after
>>> attempting to overwrite some global attributes in a Netcdf-4 classic
>>> file with HDF5 compression. Note that all three global attributes
>>> exist previously in the target file, before running the test program.
>>>
>>> After the error, the target file is corrupted. This error is
>>> strangely sensitive to the number and string lengths of the attributes
>>> to be written. Can you please tell me what the problem is? Thank
>>> you.
>>>
>>> begin
>>> outfile = "out.nc"
>>> system ("cat ozone.master.nc > " + outfile) ; create fresh writable copy
>>>
>>> out = addfile (outfile, "w")
>>> out@last_update = systemfunc ("date")
>>> out@last_update_program = "demo.global-atts.ncl version 2013-apr-01"
>>> out@last_update_source_file = "ei.ozone.1979010106.nc"
>>> print ("Update complete.")
>>> end
>>>
>>> I get the same results with NCL versions 6.0.0 and 6.1.2. My system
>>> is Mac OS 10.8.3 on a 64-bit Intel Mac desktop. Here is the console
>>> output:
>>>
>>> mac56:~/ncl/bugs/3-atts/401i 491> ncl demo.global-atts.ncl
>>> Copyright (C) 1995-2013 - All Rights Reserved
>>> University Corporation for Atmospheric Research
>>> NCAR Command Language Version 6.1.2
>>> The use of this software is governed by a License Agreement.
>>> See http://www.ncl.ucar.edu/ for more details.
>>> (0) Update complete.
>>> Segmentation fault
>>>
>>> mac56:~/ncl/bugs/3-atts/401i 492> ls -trgo
>>> total 44136
>>> -r--r--r-- 1 11294147 Apr 1 17:14 ozone.master.nc
>>> -rw-r--r-- 1 570 Apr 1 19:51 demo.global-atts.ncl
>>> -rw-r--r-- 1 11294201 Apr 1 20:16 out.nc
>>>
>>> mac56:~/ncl/bugs/3-atts/401i 493> ncdump -k ozone.master.nc
>>> netCDF-4 classic model
>>>
>>> mac56:~/ncl/bugs/3-atts/401i 494> ncdump -k out.nc
>>> ncdump: out.nc: NetCDF: Can't open HDF5 attribute
>>>
>>> mac56:~/ncl/bugs/3-atts/401i 495> uname -a
>>> Darwin mac56 12.3.0 Darwin Kernel Version 12.3.0: Sun Jan 6 22:37:10
>>> PST 2013; root:xnu-2050.22.13~1/RELEASE_X86_64 x86_64
>>>
>>> I uploaded the program and original Netcdf file (11 Mbytes) to
>>> ftp.cgd.ucar.edu/incoming, as
>>> "demo.global-atts.dave_allured.0401.zip". The original Netcdf file
>>> was also made by NCL 6.1.2 and a different script. I can also send a
>>> copy of the generating script and its input file, if you need it.
>>> Please let me know. Thanks for taking a look at this.
>>>
>>> --Dave A.
>>> NOAA/PSD/CIRES Climate Analysis Branch

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Tue Apr 2 13:28:26 2013

This archive was generated by hypermail 2.1.8 : Tue Apr 02 2013 - 21:23:48 MDT