Re: Error when writing global attributes in Netcdf-4 classic file

Oh, I see. It should be
names(i) = str_sub_str(names(i), "/", "")

Hui-Chuan

On 4/2/13 4:51 PM, Hui-Chuan Lin wrote:
> Wei,
>
> Thanks for the clarification. It's helpful.
> Below lists my observations, just for your information.
>
> a = addfile("a.nc","r")
> setfileoption("nc", "Format", "NetCDF4")
> c = addfile("ad.nc","c")
> ; copy variables
> ; this does not work -- c->/Times = a->/Times
> ;fatal:["NclNetCDF4.c":5211]:NetCDF: Name contains illegal characters
> ;fatal:["NclAdvancedFile.c":5640]:AdvancedFileAddVar: Error adding
> ;variable </Times> as type: <none>
> ; this works -- c->Times = a->/Times
> names = getfilevarnames(a)
> do i=0, dimsizes(names)-1
> ; this does not work -- str_sub_str(names(i), "/", "")
> ; this works -- replaceSingleChar(names(i), "/", "")
> c->$names(i)$ = a->$names(i)$
> end do
>
> Now a new question is how to use str_sub_str to get rid of the /
> character?
>
> Hui-Chuan
>
> On 4/2/13 3:49 PM, Wei Huang wrote:
>> Hui-Chun,
>>
>> This is kind of a feature for NetCDF4 data.
>>
>> In NetCDF4, people can use the concept of group(s),
>> which means, in a file, in can have group(s) under group(s).
>> Which is pretty like our file-system, where we have
>> directory(ies) and sub-directory(ies), and certainly files.
>>
>> So, we may have a variable T in group_A, and
>> another variable also named T in group_B.
>> Their (relative) names are the same, but they may have
>> different dimensions, or could be two totally different things.
>>
>> There will be a problem, when we have:
>>
>> f = addfile("nc4_filename.nc", "r")
>> t = f->T
>>
>> which there is ambiguity here of reading which T.
>>
>> To avoid this issue, user can read T from group_A by using:
>>
>> t = f->/group_A/T
>>
>> with this syntax, read T from root-group becomes:
>>
>> t = f->/T
>>
>> The problem here is that for lots of users, which they do not have
>> any group in their files, and this "extra" "/" cause some confusion.
>> BUT this "/" only appears for NetCDF4 files (and will appear
>> for HDF5 in 6.2.0, and probably HDF-EOS5 in 6.2.0).
>>
>> To make users code code compatible with their early code,
>> t = f->T will still work.
>>
>> But users with NetCDF4 files, which has more than one variables
>> named "T", the above code will pick the first variable named "T".
>>
>> Hope this makes things clearly.
>>
>> Thanks,
>>
>> Wei
>>
>>
>> huangwei@ucar.edu
>> VETS/CISL
>> National Center for Atmospheric Research
>> P.O. Box 3000 (1850 Table Mesa Dr.)
>> Boulder, CO 80307-3000 USA
>> (303) 497-8924
>>
>>
>>
>>
>>
>>
>> On Apr 2, 2013, at 2:58 PM, Hui-Chuan Lin <hclin@ucar.edu> wrote:
>>
>>> I've seen similar seg fault problems back in Feb on yellowstone
>>> with compressed netcdf-4 files. For example, addfile with "w"
>>> and getvaratts() didn't work properly. I came up with workarounds
>>> to do what I needed to do.
>>> After reading Dave's email, I re-did some tests.
>>> Now with 6.1.2, addfile with "w" and getvaratts() seem working,
>>> but I encountered a new problem with getfilevarnames().
>>> There is an extra / in the beginning of the variable name.
>>>
>>> A = addfile("netcdf4file.nc", "r")
>>> varnames = getfilevarnames(A)
>>> print(varnames)
>>>
>>> Variable: varnames
>>> Type: string
>>> Total Size: 1112 bytes
>>> 139 values
>>> Number of Dimensions: 1
>>> Dimensions and sizes: [139]
>>> Coordinates:
>>> (0) /Times
>>> (1) /LU_INDEX
>>> (2) /ZNU
>>> (3) /ZNW
>>>
>>> Hui-Chuan
>>>
>>> On 4/2/13 1:28 PM, Wei Huang wrote:
>>>> For tracking purpose, I have created a JIRA ticket 1692.
>>>>
>>>> Wei
>>>>
>>>> huangwei@ucar.edu
>>>> VETS/CISL
>>>> National Center for Atmospheric Research
>>>> P.O. Box 3000 (1850 Table Mesa Dr.)
>>>> Boulder, CO 80307-3000 USA
>>>> (303) 497-8924
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Apr 2, 2013, at 12:04 PM, Dave Allured - NOAA Affiliate
>>>> <dave.allured@noaa.gov> wrote:
>>>>
>>>>> Wei,
>>>>>
>>>>> Thanks for checking this out. There is definitely a problem
>>>>> somewhere. Now I am finding the same seg fault on Linux systems as
>>>>> well as different Mac OS machines, with both NCL 6.0.0 and 6.1.2. In
>>>>> fact I can not find a local system that does NOT get this seg fault.
>>>>>
>>>>> Might this be related to supplemental libraries? Following are simple
>>>>> dependency lists for both Linux and Mac. Let me know if there are any
>>>>> other diagnostics that I can provide. This is not a high priority for
>>>>> me, I have a workaround.
>>>>>
>>>>> Dave
>>>>>
>>>>> inta:~/temp/3-atts/402a 1045> date
>>>>> Tue Apr 2 11:49:25 MDT 2013
>>>>>
>>>>> inta:~/temp/3-atts/402a 1046> uname -a
>>>>> Linux inta.psd.esrl.noaa.gov 2.6.18-348.3.1.el5 #1 SMP Tue Mar 5
>>>>> 13:19:32 EST 2013 x86_64 GNU/Linux
>>>>>
>>>>> inta:~/temp/3-atts/402a 1047> echo $NCARG_ROOT
>>>>> /usr/local/ncl6.1.2
>>>>>
>>>>> inta:~/temp/3-atts/402a 1048> ls -go $NCARG_ROOT/bin/ncl
>>>>> -rwxr-xr-x 1 32917555 Feb 6 21:30 /usr/local/ncl6.1.2/bin/ncl
>>>>>
>>>>> inta:~/temp/3-atts/402a 1049> ls -go
>>>>> -rw-r--r-- 1 501 Apr 2 10:55 demo.wei-huang.0401a.ncl
>>>>> -r--r--r-- 1 11294147 Apr 1 17:14 ozone.master.nc
>>>>>
>>>>> inta:~/temp/3-atts/402a 1050> cat demo.wei-huang.0401a.ncl
>>>>> begin
>>>>> outfile = "out.nc"
>>>>> ; system ("cp ozone.master.nc " + outfile) ; create fresh
>>>>> writable copy
>>>>> system ("cat ozone.master.nc > " + outfile) ; create fresh
>>>>> writable copy
>>>>> system ("chmod +w " + outfile) ; create fresh writable copy
>>>>>
>>>>> out = addfile (outfile, "w")
>>>>>
>>>>> out@last_update = systemfunc ("date")
>>>>> out@last_update_program = "demo.global-atts.ncl version
>>>>> 2013-apr-01"
>>>>> out@last_update_source_file = "ei.ozone.1979010106.nc"
>>>>> print ("Update complete.")
>>>>> end
>>>>>
>>>>> inta:~/temp/3-atts/402a 1051> $NCARG_ROOT/bin/ncl
>>>>> demo.wei-huang.0401a.ncl
>>>>> Copyright (C) 1995-2013 - All Rights Reserved
>>>>> University Corporation for Atmospheric Research
>>>>> NCAR Command Language Version 6.1.2
>>>>> The use of this software is governed by a License Agreement.
>>>>> See http://www.ncl.ucar.edu/ for more details.
>>>>> (0) Update complete.
>>>>> Segmentation fault
>>>>>
>>>>> inta:~/temp/3-atts/402a 1052> ldd $NCARG_ROOT/bin/ncl
>>>>> linux-vdso.so.1 => (0x00007fff7dbfd000)
>>>>> libcurl.so.3 => /usr/lib64/libcurl.so.3 (0x0000003288400000)
>>>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003278600000)
>>>>> libbz2.so.1 => /usr/lib64/libbz2.so.1 (0x000000328c000000)
>>>>> libXrender.so.1 => /usr/lib64/libXrender.so.1 (0x000000327aa00000)
>>>>> libX11.so.6 => /usr/lib64/libX11.so.6 (0x0000003278e00000)
>>>>> libXext.so.6 => /usr/lib64/libXext.so.6 (0x0000003279a00000)
>>>>> libm.so.6 => /lib64/libm.so.6 (0x0000003277a00000)
>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x0000003277e00000)
>>>>> libstdc++.so.6 => /usr/local/gcc-4.7/lib/libstdc++.so.6
>>>>> (0x00002b9f6bc6c000)
>>>>> libgcc_s.so.1 => /usr/local/gcc-4.7/lib/libgcc_s.so.1
>>>>> (0x00002b9f6bf76000)
>>>>> libc.so.6 => /lib64/libc.so.6 (0x0000003277600000)
>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003277200000)
>>>>> libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2
>>>>> (0x0000003282c00000)
>>>>> libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x0000003282400000)
>>>>> libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3
>>>>> (0x0000003282000000)
>>>>> libcom_err.so.2 => /lib64/libcom_err.so.2 (0x0000003281c00000)
>>>>> libidn.so.11 => /usr/lib64/libidn.so.11 (0x0000003284000000)
>>>>> libssl.so.6 => /lib64/libssl.so.6 (0x0000003288c00000)
>>>>> libcrypto.so.6 => /lib64/libcrypto.so.6 (0x0000003288800000)
>>>>> libz.so.1 => /lib64/libz.so.1 (0x0000003278200000)
>>>>> libXau.so.6 => /usr/lib64/libXau.so.6 (0x0000003279600000)
>>>>> libXdmcp.so.6 => /usr/lib64/libXdmcp.so.6 (0x0000003278a00000)
>>>>> libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0
>>>>> (0x0000003282800000)
>>>>> libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x0000003281400000)
>>>>> libresolv.so.2 => /lib64/libresolv.so.2 (0x0000003280c00000)
>>>>> libselinux.so.1 => /lib64/libselinux.so.1 (0x000000327d200000)
>>>>> libsepol.so.1 => /lib64/libsepol.so.1 (0x000000327ce00000)
>>>>>
>>>>> ========================================================
>>>>>
>>>>> mac56:~/ncl/bugs/3-atts/402a 35> uname -a
>>>>> Darwin mac56 12.3.0 Darwin Kernel Version 12.3.0: Sun Jan 6 22:37:10
>>>>> PST 2013; root:xnu-2050.22.13~1/RELEASE_X86_64 x86_64
>>>>>
>>>>> mac56:~/ncl/bugs/3-atts/402a 36> echo $NCARG_ROOT
>>>>> /usr/local/ncl6.1.2
>>>>>
>>>>> mac56:~/ncl/bugs/3-atts/402a 37> ls -dgo $NCARG_ROOT/bin/ncl
>>>>> -rwxr-xr-x 1 33112624 Feb 6 21:38 /usr/local/ncl6.1.2/bin/ncl
>>>>>
>>>>> mac56:~/ncl/bugs/3-atts/402a 38> ldd $NCARG_ROOT/bin/ncl
>>>>> /usr/local/ncl6.1.2/bin/ncl:
>>>>> /usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current
>>>>> version 7.0.0)
>>>>> /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current
>>>>> version 7.0.0)
>>>>> /usr/X11/lib/libXrender.1.dylib (compatibility version 5.0.0,
>>>>> current
>>>>> version 5.0.0)
>>>>> /usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current
>>>>> version 1.0.5)
>>>>> /usr/X11/lib/libXpm.4.dylib (compatibility version 16.0.0, current
>>>>> version 16.0.0)
>>>>> /usr/X11/lib/libX11.6.dylib (compatibility version 10.0.0, current
>>>>> version 10.0.0)
>>>>> /usr/X11/lib/libXext.6.dylib (compatibility version 11.0.0,
>>>>> current
>>>>> version 11.0.0)
>>>>> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
>>>>> version 159.1.0)
>>>>> /usr/local/lib/libstdc++.6.dylib (compatibility version 7.0.0,
>>>>> current version 7.14.0)
>>>>> /usr/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0,
>>>>> current
>>>>> version 1.0.0)
>>>>>
>>>>> On Mon, Apr 1, 2013 at 9:35 PM, Wei Huang <huangwei@ucar.edu> wrote:
>>>>>> Dave,
>>>>>>
>>>>>> I can not reproduce your problem.
>>>>>>
>>>>>> I only added:
>>>>>> system ("chmod +w " + outfile)
>>>>>> after system call in your script (as the original file is write
>>>>>> protected).
>>>>>>
>>>>>> and it runs fine.
>>>>>>
>>>>>> I tested on Mac, with osx 10.8.3.
>>>>>>
>>>>>> Here is the full script I tested:
>>>>>>
>>>>>> ; 2013-apr-01 Test program reduced from era-update.ncl.
>>>>>> ; Demonstrates seg fault when trying to overwrite
>>>>>> global
>>>>>> ; attributes in a Netcdf-4 classic file.
>>>>>> ; By Dave Allured, NOAA/PSD/CIRES Climate Analysis
>>>>>> Branch.
>>>>>>
>>>>>> begin
>>>>>> outfile = "out.nc"
>>>>>> ; system ("cp ozone.master.nc " + outfile) ; create fresh
>>>>>> writable copy
>>>>>> system ("cat ozone.master.nc > " + outfile) ; create fresh
>>>>>> writable copy
>>>>>> system ("chmod +w " + outfile) ; create fresh writable copy
>>>>>>
>>>>>> out = addfile (outfile, "w")
>>>>>>
>>>>>> out@last_update = systemfunc ("date")
>>>>>> out@last_update_program = "demo.global-atts.ncl version
>>>>>> 2013-apr-01"
>>>>>> out@last_update_source_file = "ei.ozone.1979010106.nc"
>>>>>> print ("Update complete.")
>>>>>> end
>>>>>>
>>>>>>
>>>>>> and part of ncdump -h out.nc:
>>>>>>
>>>>>> // global attributes:
>>>>>> :title = "ERA-Interim Ozone mass mixing ratio, 4
>>>>>> times daily" ;
>>>>>> :history = "Mon Apr 1 17:14:49 MDT 2013:\n",
>>>>>> "Initial file made by era-create.ncl
>>>>>> version 2013-apr-01 TEST" ;
>>>>>> :dataset = "ERA-Interim" ;
>>>>>> :version = "2.0" ;
>>>>>> :references =
>>>>>> "http://www.ecmwf.int/research/era/do/get/era-interim;\n",
>>>>>> "D. P. Dee et al (2011), The ERA-Interim
>>>>>> reanalysis:\n",
>>>>>> " configuration and performance of the
>>>>>> data assimilation system,\n",
>>>>>> " Q.J.R. Meteorol. Soc., Vol. 137:
>>>>>> 553-597, DOI: 10.1002/qj.828" ;
>>>>>> :center = "European Center for Medium-Range
>>>>>> Weather Forecasts (RSMC)" ;
>>>>>> :metadata_source_file =
>>>>>> "../ei.oper.an.pl.regn128sc.1979010100" ;
>>>>>> :Conventions = "CF-1.6" ;
>>>>>> :last_update = "Mon Apr 1 21:28:20 MDT 2013" ;
>>>>>> :last_update_program = "demo.global-atts.ncl
>>>>>> version 2013-apr-01" ;
>>>>>> :last_update_source_file = "ei.ozone.1979010106.nc" ;
>>>>>> }
>>>>>>
>>>>>>
>>>>>> Wei
>>>>>>
>>>>>> huangwei@ucar.edu
>>>>>> VETS/CISL
>>>>>> National Center for Atmospheric Research
>>>>>> P.O. Box 3000 (1850 Table Mesa Dr.)
>>>>>> Boulder, CO 80307-3000 USA
>>>>>> (303) 497-8924
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Apr 1, 2013, at 8:41 PM, Dave Allured - NOAA Affiliate
>>>>>> <dave.allured@noaa.gov> wrote:
>>>>>>
>>>>>>> NCL support team,
>>>>>>>
>>>>>>> This program results in a seg fault error on program exit, after
>>>>>>> attempting to overwrite some global attributes in a Netcdf-4 classic
>>>>>>> file with HDF5 compression. Note that all three global attributes
>>>>>>> exist previously in the target file, before running the test
>>>>>>> program.
>>>>>>>
>>>>>>> After the error, the target file is corrupted. This error is
>>>>>>> strangely sensitive to the number and string lengths of the
>>>>>>> attributes
>>>>>>> to be written. Can you please tell me what the problem is? Thank
>>>>>>> you.
>>>>>>>
>>>>>>> begin
>>>>>>> outfile = "out.nc"
>>>>>>> system ("cat ozone.master.nc > " + outfile) ; create fresh
>>>>>>> writable copy
>>>>>>>
>>>>>>> out = addfile (outfile, "w")
>>>>>>> out@last_update = systemfunc ("date")
>>>>>>> out@last_update_program = "demo.global-atts.ncl version 2013-apr-01"
>>>>>>> out@last_update_source_file = "ei.ozone.1979010106.nc"
>>>>>>> print ("Update complete.")
>>>>>>> end
>>>>>>>
>>>>>>> I get the same results with NCL versions 6.0.0 and 6.1.2. My system
>>>>>>> is Mac OS 10.8.3 on a 64-bit Intel Mac desktop. Here is the console
>>>>>>> output:
>>>>>>>
>>>>>>> mac56:~/ncl/bugs/3-atts/401i 491> ncl demo.global-atts.ncl
>>>>>>> Copyright (C) 1995-2013 - All Rights Reserved
>>>>>>> University Corporation for Atmospheric Research
>>>>>>> NCAR Command Language Version 6.1.2
>>>>>>> The use of this software is governed by a License Agreement.
>>>>>>> See http://www.ncl.ucar.edu/ for more details.
>>>>>>> (0) Update complete.
>>>>>>> Segmentation fault
>>>>>>>
>>>>>>> mac56:~/ncl/bugs/3-atts/401i 492> ls -trgo
>>>>>>> total 44136
>>>>>>> -r--r--r-- 1 11294147 Apr 1 17:14 ozone.master.nc
>>>>>>> -rw-r--r-- 1 570 Apr 1 19:51 demo.global-atts.ncl
>>>>>>> -rw-r--r-- 1 11294201 Apr 1 20:16 out.nc
>>>>>>>
>>>>>>> mac56:~/ncl/bugs/3-atts/401i 493> ncdump -k ozone.master.nc
>>>>>>> netCDF-4 classic model
>>>>>>>
>>>>>>> mac56:~/ncl/bugs/3-atts/401i 494> ncdump -k out.nc
>>>>>>> ncdump: out.nc: NetCDF: Can't open HDF5 attribute
>>>>>>>
>>>>>>> mac56:~/ncl/bugs/3-atts/401i 495> uname -a
>>>>>>> Darwin mac56 12.3.0 Darwin Kernel Version 12.3.0: Sun Jan 6
>>>>>>> 22:37:10
>>>>>>> PST 2013; root:xnu-2050.22.13~1/RELEASE_X86_64 x86_64
>>>>>>>
>>>>>>> I uploaded the program and original Netcdf file (11 Mbytes) to
>>>>>>> ftp.cgd.ucar.edu/incoming, as
>>>>>>> "demo.global-atts.dave_allured.0401.zip". The original Netcdf file
>>>>>>> was also made by NCL 6.1.2 and a different script. I can also
>>>>>>> send a
>>>>>>> copy of the generating script and its input file, if you need it.
>>>>>>> Please let me know. Thanks for taking a look at this.
>>>>>>>
>>>>>>> --Dave A.
>>>>>>> NOAA/PSD/CIRES Climate Analysis Branch
>>>>
>>>> _______________________________________________
>>>> ncl-talk mailing list
>>>> List instructions, subscriber options, unsubscribe:
>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>
>>>
>>> _______________________________________________
>>> ncl-talk mailing list
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Tue Apr 2 17:29:57 2013