Re: Issue with writing large data out to netCDF

Interesting. I forgot to include the link to the example I mentioned (now
below), which uses simply the NetCDF4 option, albeit for a single large
variable.

http://ncl.ucar.edu/Applications/write_nc4_1dUnlimited.shtml

I suppose my assumption that there wasn't a difference between "NetCDF4"
and "NetCDF4Classic" was not accurate, although the discussion under the
Format subsection of the setfileoption documentation isn't particularly
clear about the differences (see
http://www.ncl.ucar.edu/Document/Functions/Built-in/setfileoption.shtml )
and implies that both should function just fine with unlimited variables
>2GB.

I'll give Wei's suggestions a try - I can already state that the first
suggestion (just changing fout to "nc") did not work for me yesterday. Per
a section of the setfileoption documentation...

"If the first argument is a string specifying a format, then the option
setting applies to any file of the specified format when it is first opened
using *addfile*<http://www.ncl.ucar.edu/Document/Functions/Built-in/addfile.shtml>
 or *addfiles
<http://www.ncl.ucar.edu/Document/Functions/Built-in/addfiles.shtml>.
*... Files
referenced by existing file variables will not be affected. On the other
hand, options set using a file variable as the first argument affect only
the specified file, and may alter NCL's treatment of it dynamically."

...would it be more logical to place setfileoption("nc",...,...) statements
before the addfile call and setfileoption(fout,...,...) styled-statements
after a file is open?

Kyle

----------------------------------------
Kyle S. Griffin
Department of Atmospheric and Oceanic Sciences
University of Wisconsin - Madison
Room 1421
1225 W Dayton St, Madison, WI 53706
Email: ksgriffin2@wisc.edu

On Fri, Feb 14, 2014 at 10:28 AM, Wei Huang <huangwei@ucar.edu> wrote:

> Kyle,
>
> You may try change line:
>
> setfileoption(fout,"format","netcdf4")
>
> to:
>
> setfileoption("nc","format","netcdf4")
>
> If that did not solve your issues,
> then add this line (just below the above line):
> setfileoption("nc", "FileStrucuture", "Advanced")
>
> If both failed, let ncl-talk know.
>
> Thanks,
>
> Wei
>
>
> huangwei@ucar.edu
> VETS/CISL
> National Center for Atmospheric Research
> P.O. Box 3000 (1850 Table Mesa Dr.)
> Boulder, CO 80307-3000 USA
> (303) 497-8924
>
>
>
>
>
>
> On Feb 14, 2014, at 8:08 AM, Kyle Griffin <ksgriffin2@wisc.edu> wrote:
>
> Hi all,
> I've recently been having some issues writing out large variables (>2GB)
> to a netCDF file and I'm not sure why these limits are present. Via some
> testing, I was able to isolate that the problem does seem to occur when the
> individual variables cross that magical 2 GB threshold. This code is
> essentially identical to that which has been used on a number of other
> occasions to write large (>10GB) variables to netCDF, but with only one
> variable in a file. I was also defining "netcdf4classic" in my previous
> writing-large-variable experience, but I don't believe this should make a
> difference and the example below is almost entirely in-line with the
> example on writing large variables.
>
> Any advice on where my code is going wrong? A bit more info on the sizes
> of the variables is below. Thanks for any help.
>
>
> Kyle
>
> Output:
>
> fatal:NetCDF: Operation not allowed in define mode: error attempting to
> write variable (sign) to file
> (/glade/scratch/kgriffin/dp/TEST_dp_stddev_1201.nc)
> fatal:["Execute.c":8128]:Execute: Error occurred at or near line 234 in
> file ./two-way-climo.ncl
> ;.....code continues past this, and then dies at:...........
> ncendef: ncid 131072: NetCDF: One or more variable sizes violate format
> constraints
> ncclose: ncid 131072: NetCDF: One or more variable sizes violate format
> constraints
>
>
> Variables:
>
> sign, time, lev, biglev, lat, lon are all 1-D coordinate variables.
> mean and sddev are of size [2 x 124 x 16 x 361 x 720] and type float -
> roughly 3.8GB.
> bigmean and bigsddev are the same size, but with only 3 instead of 16 for
> the third dimension.
>
>
> File-writing code:
>
> system("rm -f "+outfile)
> fout = addfile (outfile, "c")
> setfileoption(fout,"format","netcdf4")
> ;setfileoption(fout,"DefineMode",True)
> fAtt = True
> fAtt@creation_date = systemfunc ("date")
> fileattdef( fout, fAtt )
>
> dimNames = (/"sign","time","lev","biglev","lat", "lon"/)
> dimSizes =
> (/dimsizes(climo&sign),dimsizes(climo&time),dimsizes(climo&lev),dimsizes(bigclimo&biglev),dimsizes(climo&lat),
> dimsizes(climo&lon) /)
> dimUnlim = (/False,False,False,False,False, False/)
> filedimdef(fout,dimNames,dimSizes,dimUnlim)
>
> filevardef(fout, "sign" ,typeof(climo&sign),getvardims(climo&sign))
> filevardef(fout, "time" ,typeof(climo&time),getvardims(climo&time))
> filevardef(fout, "lev" ,typeof(climo&lev),getvardims(climo&lev))
> filevardef(fout, "biglev"
> ,typeof(bigclimo&biglev),getvardims(bigclimo&biglev))
> filevardef(fout, "lat" ,typeof(climo&lat),getvardims(climo&lat))
> filevardef(fout, "lon" ,typeof(climo&lon),getvardims(climo&lon))
> filevardef(fout, "mean",typeof(mean),getvardims(mean))
> filevardef(fout, "bigmean",typeof(bigmean),getvardims(bigmean))
> filevardef(fout, "sddev",typeof(climo),getvardims(climo))
> filevardef(fout, "bigsddev",typeof(bigclimo),getvardims(bigclimo))
>
> filevarattdef(fout,"time" ,climo&time)
> filevarattdef(fout,"lev" ,climo&lev)
> filevarattdef(fout,"biglev",bigclimo&biglev)
> filevarattdef(fout,"lat" ,climo&lat)
> filevarattdef(fout,"lon" ,climo&lon)
> filevarattdef(fout,"mean",mean)
> filevarattdef(fout,"bigmean",bigmean)
> filevarattdef(fout,"sddev",climo)
> filevarattdef(fout,"bigsddev",bigclimo)
>
> ;setfileoption(fout,"DefineMode",False)
> print("Predefining done. Writing now.")
>
> fout->sign = (/climo&sign/)
> fout->time = (/climo&time/)
> fout->lev = (/climo&lev/)
> fout->biglev = (/bigclimo&biglev/)
> fout->lat = (/climo&lat/)
> fout->lon = (/climo&lon/)
> fout->mean = (/mean/)
> fout->bigmean = (/bigmean/)
> fout->sddev = (/climo/)
> fout->bigsddev = (/bigclimo/)
>
>
> ----------------------------------------
> Kyle S. Griffin
> Department of Atmospheric and Oceanic Sciences
> University of Wisconsin - Madison
> Room 1421
> 1225 W Dayton St, Madison, WI 53706
> Email: ksgriffin2@wisc.edu
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
>

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Fri Feb 14 09:37:49 2014