Re: Error when writing global attributes in Netcdf-4 classic file

From: Wei Huang <huangwei_at_nyahnyahspammersnyahnyah>
Date: Tue Apr 02 2013 - 15:49:04 MDT

Hui-Chun,

This is kind of a feature for NetCDF4 data.

In NetCDF4, people can use the concept of group(s),
which means, in a file, in can have group(s) under group(s).
Which is pretty like our file-system, where we have
directory(ies) and sub-directory(ies), and certainly files.

So, we may have a variable T in group_A, and
another variable also named T in group_B.
Their (relative) names are the same, but they may have
different dimensions, or could be two totally different things.

There will be a problem, when we have:

 f = addfile("nc4_filename.nc", "r")
 t = f->T

which there is ambiguity here of reading which T.

To avoid this issue, user can read T from group_A by using:

t = f->/group_A/T

with this syntax, read T from root-group becomes:

t = f->/T

The problem here is that for lots of users, which they do not have
any group in their files, and this "extra" "/" cause some confusion.
BUT this "/" only appears for NetCDF4 files (and will appear
for HDF5 in 6.2.0, and probably HDF-EOS5 in 6.2.0).

To make users code code compatible with their early code,
t = f->T will still work.

But users with NetCDF4 files, which has more than one variables
named "T", the above code will pick the first variable named "T".

Hope this makes things clearly.

Thanks,

Wei

huangwei@ucar.edu
VETS/CISL
National Center for Atmospheric Research
P.O. Box 3000 (1850 Table Mesa Dr.)
Boulder, CO 80307-3000 USA
(303) 497-8924

On Apr 2, 2013, at 2:58 PM, Hui-Chuan Lin <hclin@ucar.edu> wrote:

> I've seen similar seg fault problems back in Feb on yellowstone
> with compressed netcdf-4 files. For example, addfile with "w"
> and getvaratts() didn't work properly. I came up with workarounds
> to do what I needed to do.
> After reading Dave's email, I re-did some tests.
> Now with 6.1.2, addfile with "w" and getvaratts() seem working,
> but I encountered a new problem with getfilevarnames().
> There is an extra / in the beginning of the variable name.
>
> A = addfile("netcdf4file.nc", "r")
> varnames = getfilevarnames(A)
> print(varnames)
>
> Variable: varnames
> Type: string
> Total Size: 1112 bytes
> 139 values
> Number of Dimensions: 1
> Dimensions and sizes: [139]
> Coordinates:
> (0) /Times
> (1) /LU_INDEX
> (2) /ZNU
> (3) /ZNW
>
> Hui-Chuan
>
> On 4/2/13 1:28 PM, Wei Huang wrote:
>> For tracking purpose, I have created a JIRA ticket 1692.
>>
>> Wei
>>
>> huangwei@ucar.edu
>> VETS/CISL
>> National Center for Atmospheric Research
>> P.O. Box 3000 (1850 Table Mesa Dr.)
>> Boulder, CO 80307-3000 USA
>> (303) 497-8924
>>
>>
>>
>>
>>
>>
>> On Apr 2, 2013, at 12:04 PM, Dave Allured - NOAA Affiliate <dave.allured@noaa.gov> wrote:
>>
>>> Wei,
>>>
>>> Thanks for checking this out. There is definitely a problem
>>> somewhere. Now I am finding the same seg fault on Linux systems as
>>> well as different Mac OS machines, with both NCL 6.0.0 and 6.1.2. In
>>> fact I can not find a local system that does NOT get this seg fault.
>>>
>>> Might this be related to supplemental libraries? Following are simple
>>> dependency lists for both Linux and Mac. Let me know if there are any
>>> other diagnostics that I can provide. This is not a high priority for
>>> me, I have a workaround.
>>>
>>> Dave
>>>
>>> inta:~/temp/3-atts/402a 1045> date
>>> Tue Apr 2 11:49:25 MDT 2013
>>>
>>> inta:~/temp/3-atts/402a 1046> uname -a
>>> Linux inta.psd.esrl.noaa.gov 2.6.18-348.3.1.el5 #1 SMP Tue Mar 5
>>> 13:19:32 EST 2013 x86_64 GNU/Linux
>>>
>>> inta:~/temp/3-atts/402a 1047> echo $NCARG_ROOT
>>> /usr/local/ncl6.1.2
>>>
>>> inta:~/temp/3-atts/402a 1048> ls -go $NCARG_ROOT/bin/ncl
>>> -rwxr-xr-x 1 32917555 Feb 6 21:30 /usr/local/ncl6.1.2/bin/ncl
>>>
>>> inta:~/temp/3-atts/402a 1049> ls -go
>>> -rw-r--r-- 1 501 Apr 2 10:55 demo.wei-huang.0401a.ncl
>>> -r--r--r-- 1 11294147 Apr 1 17:14 ozone.master.nc
>>>
>>> inta:~/temp/3-atts/402a 1050> cat demo.wei-huang.0401a.ncl
>>> begin
>>> outfile = "out.nc"
>>> ; system ("cp ozone.master.nc " + outfile) ; create fresh writable copy
>>> system ("cat ozone.master.nc > " + outfile) ; create fresh writable copy
>>> system ("chmod +w " + outfile) ; create fresh writable copy
>>>
>>> out = addfile (outfile, "w")
>>>
>>> out@last_update = systemfunc ("date")
>>> out@last_update_program = "demo.global-atts.ncl version 2013-apr-01"
>>> out@last_update_source_file = "ei.ozone.1979010106.nc"
>>> print ("Update complete.")
>>> end
>>>
>>> inta:~/temp/3-atts/402a 1051> $NCARG_ROOT/bin/ncl demo.wei-huang.0401a.ncl
>>> Copyright (C) 1995-2013 - All Rights Reserved
>>> University Corporation for Atmospheric Research
>>> NCAR Command Language Version 6.1.2
>>> The use of this software is governed by a License Agreement.
>>> See http://www.ncl.ucar.edu/ for more details.
>>> (0) Update complete.
>>> Segmentation fault
>>>
>>> inta:~/temp/3-atts/402a 1052> ldd $NCARG_ROOT/bin/ncl
>>> linux-vdso.so.1 => (0x00007fff7dbfd000)
>>> libcurl.so.3 => /usr/lib64/libcurl.so.3 (0x0000003288400000)
>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003278600000)
>>> libbz2.so.1 => /usr/lib64/libbz2.so.1 (0x000000328c000000)
>>> libXrender.so.1 => /usr/lib64/libXrender.so.1 (0x000000327aa00000)
>>> libX11.so.6 => /usr/lib64/libX11.so.6 (0x0000003278e00000)
>>> libXext.so.6 => /usr/lib64/libXext.so.6 (0x0000003279a00000)
>>> libm.so.6 => /lib64/libm.so.6 (0x0000003277a00000)
>>> libdl.so.2 => /lib64/libdl.so.2 (0x0000003277e00000)
>>> libstdc++.so.6 => /usr/local/gcc-4.7/lib/libstdc++.so.6 (0x00002b9f6bc6c000)
>>> libgcc_s.so.1 => /usr/local/gcc-4.7/lib/libgcc_s.so.1 (0x00002b9f6bf76000)
>>> libc.so.6 => /lib64/libc.so.6 (0x0000003277600000)
>>> /lib64/ld-linux-x86-64.so.2 (0x0000003277200000)
>>> libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x0000003282c00000)
>>> libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x0000003282400000)
>>> libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x0000003282000000)
>>> libcom_err.so.2 => /lib64/libcom_err.so.2 (0x0000003281c00000)
>>> libidn.so.11 => /usr/lib64/libidn.so.11 (0x0000003284000000)
>>> libssl.so.6 => /lib64/libssl.so.6 (0x0000003288c00000)
>>> libcrypto.so.6 => /lib64/libcrypto.so.6 (0x0000003288800000)
>>> libz.so.1 => /lib64/libz.so.1 (0x0000003278200000)
>>> libXau.so.6 => /usr/lib64/libXau.so.6 (0x0000003279600000)
>>> libXdmcp.so.6 => /usr/lib64/libXdmcp.so.6 (0x0000003278a00000)
>>> libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x0000003282800000)
>>> libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x0000003281400000)
>>> libresolv.so.2 => /lib64/libresolv.so.2 (0x0000003280c00000)
>>> libselinux.so.1 => /lib64/libselinux.so.1 (0x000000327d200000)
>>> libsepol.so.1 => /lib64/libsepol.so.1 (0x000000327ce00000)
>>>
>>> ========================================================
>>>
>>> mac56:~/ncl/bugs/3-atts/402a 35> uname -a
>>> Darwin mac56 12.3.0 Darwin Kernel Version 12.3.0: Sun Jan 6 22:37:10
>>> PST 2013; root:xnu-2050.22.13~1/RELEASE_X86_64 x86_64
>>>
>>> mac56:~/ncl/bugs/3-atts/402a 36> echo $NCARG_ROOT
>>> /usr/local/ncl6.1.2
>>>
>>> mac56:~/ncl/bugs/3-atts/402a 37> ls -dgo $NCARG_ROOT/bin/ncl
>>> -rwxr-xr-x 1 33112624 Feb 6 21:38 /usr/local/ncl6.1.2/bin/ncl
>>>
>>> mac56:~/ncl/bugs/3-atts/402a 38> ldd $NCARG_ROOT/bin/ncl
>>> /usr/local/ncl6.1.2/bin/ncl:
>>> /usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 7.0.0)
>>> /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
>>> /usr/X11/lib/libXrender.1.dylib (compatibility version 5.0.0, current
>>> version 5.0.0)
>>> /usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.5)
>>> /usr/X11/lib/libXpm.4.dylib (compatibility version 16.0.0, current
>>> version 16.0.0)
>>> /usr/X11/lib/libX11.6.dylib (compatibility version 10.0.0, current
>>> version 10.0.0)
>>> /usr/X11/lib/libXext.6.dylib (compatibility version 11.0.0, current
>>> version 11.0.0)
>>> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
>>> version 159.1.0)
>>> /usr/local/lib/libstdc++.6.dylib (compatibility version 7.0.0,
>>> current version 7.14.0)
>>> /usr/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current
>>> version 1.0.0)
>>>
>>> On Mon, Apr 1, 2013 at 9:35 PM, Wei Huang <huangwei@ucar.edu> wrote:
>>>> Dave,
>>>>
>>>> I can not reproduce your problem.
>>>>
>>>> I only added:
>>>> system ("chmod +w " + outfile)
>>>> after system call in your script (as the original file is write protected).
>>>>
>>>> and it runs fine.
>>>>
>>>> I tested on Mac, with osx 10.8.3.
>>>>
>>>> Here is the full script I tested:
>>>>
>>>> ; 2013-apr-01 Test program reduced from era-update.ncl.
>>>> ; Demonstrates seg fault when trying to overwrite global
>>>> ; attributes in a Netcdf-4 classic file.
>>>> ; By Dave Allured, NOAA/PSD/CIRES Climate Analysis Branch.
>>>>
>>>> begin
>>>> outfile = "out.nc"
>>>> ; system ("cp ozone.master.nc " + outfile) ; create fresh writable copy
>>>> system ("cat ozone.master.nc > " + outfile) ; create fresh writable copy
>>>> system ("chmod +w " + outfile) ; create fresh writable copy
>>>>
>>>> out = addfile (outfile, "w")
>>>>
>>>> out@last_update = systemfunc ("date")
>>>> out@last_update_program = "demo.global-atts.ncl version 2013-apr-01"
>>>> out@last_update_source_file = "ei.ozone.1979010106.nc"
>>>> print ("Update complete.")
>>>> end
>>>>
>>>>
>>>> and part of ncdump -h out.nc:
>>>>
>>>> // global attributes:
>>>> :title = "ERA-Interim Ozone mass mixing ratio, 4 times daily" ;
>>>> :history = "Mon Apr 1 17:14:49 MDT 2013:\n",
>>>> "Initial file made by era-create.ncl version 2013-apr-01 TEST" ;
>>>> :dataset = "ERA-Interim" ;
>>>> :version = "2.0" ;
>>>> :references = "http://www.ecmwf.int/research/era/do/get/era-interim;\n",
>>>> "D. P. Dee et al (2011), The ERA-Interim reanalysis:\n",
>>>> " configuration and performance of the data assimilation system,\n",
>>>> " Q.J.R. Meteorol. Soc., Vol. 137: 553-597, DOI: 10.1002/qj.828" ;
>>>> :center = "European Center for Medium-Range Weather Forecasts (RSMC)" ;
>>>> :metadata_source_file = "../ei.oper.an.pl.regn128sc.1979010100" ;
>>>> :Conventions = "CF-1.6" ;
>>>> :last_update = "Mon Apr 1 21:28:20 MDT 2013" ;
>>>> :last_update_program = "demo.global-atts.ncl version 2013-apr-01" ;
>>>> :last_update_source_file = "ei.ozone.1979010106.nc" ;
>>>> }
>>>>
>>>>
>>>> Wei
>>>>
>>>> huangwei@ucar.edu
>>>> VETS/CISL
>>>> National Center for Atmospheric Research
>>>> P.O. Box 3000 (1850 Table Mesa Dr.)
>>>> Boulder, CO 80307-3000 USA
>>>> (303) 497-8924
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Apr 1, 2013, at 8:41 PM, Dave Allured - NOAA Affiliate <dave.allured@noaa.gov> wrote:
>>>>
>>>>> NCL support team,
>>>>>
>>>>> This program results in a seg fault error on program exit, after
>>>>> attempting to overwrite some global attributes in a Netcdf-4 classic
>>>>> file with HDF5 compression. Note that all three global attributes
>>>>> exist previously in the target file, before running the test program.
>>>>>
>>>>> After the error, the target file is corrupted. This error is
>>>>> strangely sensitive to the number and string lengths of the attributes
>>>>> to be written. Can you please tell me what the problem is? Thank
>>>>> you.
>>>>>
>>>>> begin
>>>>> outfile = "out.nc"
>>>>> system ("cat ozone.master.nc > " + outfile) ; create fresh writable copy
>>>>>
>>>>> out = addfile (outfile, "w")
>>>>> out@last_update = systemfunc ("date")
>>>>> out@last_update_program = "demo.global-atts.ncl version 2013-apr-01"
>>>>> out@last_update_source_file = "ei.ozone.1979010106.nc"
>>>>> print ("Update complete.")
>>>>> end
>>>>>
>>>>> I get the same results with NCL versions 6.0.0 and 6.1.2. My system
>>>>> is Mac OS 10.8.3 on a 64-bit Intel Mac desktop. Here is the console
>>>>> output:
>>>>>
>>>>> mac56:~/ncl/bugs/3-atts/401i 491> ncl demo.global-atts.ncl
>>>>> Copyright (C) 1995-2013 - All Rights Reserved
>>>>> University Corporation for Atmospheric Research
>>>>> NCAR Command Language Version 6.1.2
>>>>> The use of this software is governed by a License Agreement.
>>>>> See http://www.ncl.ucar.edu/ for more details.
>>>>> (0) Update complete.
>>>>> Segmentation fault
>>>>>
>>>>> mac56:~/ncl/bugs/3-atts/401i 492> ls -trgo
>>>>> total 44136
>>>>> -r--r--r-- 1 11294147 Apr 1 17:14 ozone.master.nc
>>>>> -rw-r--r-- 1 570 Apr 1 19:51 demo.global-atts.ncl
>>>>> -rw-r--r-- 1 11294201 Apr 1 20:16 out.nc
>>>>>
>>>>> mac56:~/ncl/bugs/3-atts/401i 493> ncdump -k ozone.master.nc
>>>>> netCDF-4 classic model
>>>>>
>>>>> mac56:~/ncl/bugs/3-atts/401i 494> ncdump -k out.nc
>>>>> ncdump: out.nc: NetCDF: Can't open HDF5 attribute
>>>>>
>>>>> mac56:~/ncl/bugs/3-atts/401i 495> uname -a
>>>>> Darwin mac56 12.3.0 Darwin Kernel Version 12.3.0: Sun Jan 6 22:37:10
>>>>> PST 2013; root:xnu-2050.22.13~1/RELEASE_X86_64 x86_64
>>>>>
>>>>> I uploaded the program and original Netcdf file (11 Mbytes) to
>>>>> ftp.cgd.ucar.edu/incoming, as
>>>>> "demo.global-atts.dave_allured.0401.zip". The original Netcdf file
>>>>> was also made by NCL 6.1.2 and a different script. I can also send a
>>>>> copy of the generating script and its input file, if you need it.
>>>>> Please let me know. Thanks for taking a look at this.
>>>>>
>>>>> --Dave A.
>>>>> NOAA/PSD/CIRES Climate Analysis Branch
>>
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Tue Apr 2 15:49:18 2013

This archive was generated by hypermail 2.1.8 : Tue Apr 02 2013 - 21:23:48 MDT