Re: Issue with writing large data out to netCDF

Kyle,

You may try change line:

setfileoption(fout,"format","netcdf4")

to:

setfileoption("nc","format","netcdf4")

If that did not solve your issues,
then add this line (just below the above line):
setfileoption("nc", "FileStrucuture", "Advanced")

If both failed, let ncl-talk know.

Thanks,

Wei

huangwei@ucar.edu
VETS/CISL
National Center for Atmospheric Research
P.O. Box 3000 (1850 Table Mesa Dr.)
Boulder, CO 80307-3000 USA
(303) 497-8924

On Feb 14, 2014, at 8:08 AM, Kyle Griffin <ksgriffin2@wisc.edu> wrote:

> Hi all,
>
> I've recently been having some issues writing out large variables (>2GB) to a netCDF file and I'm not sure why these limits are present. Via some testing, I was able to isolate that the problem does seem to occur when the individual variables cross that magical 2 GB threshold. This code is essentially identical to that which has been used on a number of other occasions to write large (>10GB) variables to netCDF, but with only one variable in a file. I was also defining "netcdf4classic" in my previous writing-large-variable experience, but I don't believe this should make a difference and the example below is almost entirely in-line with the example on writing large variables.
>
> Any advice on where my code is going wrong? A bit more info on the sizes of the variables is below. Thanks for any help.
>
>
> Kyle
>
> Output:
>
> fatal:NetCDF: Operation not allowed in define mode: error attempting to write variable (sign) to file (/glade/scratch/kgriffin/dp/TEST_dp_stddev_1201.nc)
> fatal:["Execute.c":8128]:Execute: Error occurred at or near line 234 in file ./two-way-climo.ncl
> ;.....code continues past this, and then dies at:...........
> ncendef: ncid 131072: NetCDF: One or more variable sizes violate format constraints
> ncclose: ncid 131072: NetCDF: One or more variable sizes violate format constraints
>
>
> Variables:
>
> sign, time, lev, biglev, lat, lon are all 1-D coordinate variables.
> mean and sddev are of size [2 x 124 x 16 x 361 x 720] and type float - roughly 3.8GB.
> bigmean and bigsddev are the same size, but with only 3 instead of 16 for the third dimension.
>
>
> File-writing code:
>
> system("rm -f "+outfile)
> fout = addfile (outfile, "c")
> setfileoption(fout,"format","netcdf4")
> ;setfileoption(fout,"DefineMode",True)
> fAtt = True
> fAtt@creation_date = systemfunc ("date")
> fileattdef( fout, fAtt )
>
> dimNames = (/"sign","time","lev","biglev","lat", "lon"/)
> dimSizes = (/dimsizes(climo&sign),dimsizes(climo&time),dimsizes(climo&lev),dimsizes(bigclimo&biglev),dimsizes(climo&lat), dimsizes(climo&lon) /)
> dimUnlim = (/False,False,False,False,False, False/)
> filedimdef(fout,dimNames,dimSizes,dimUnlim)
>
> filevardef(fout, "sign" ,typeof(climo&sign),getvardims(climo&sign))
> filevardef(fout, "time" ,typeof(climo&time),getvardims(climo&time))
> filevardef(fout, "lev" ,typeof(climo&lev),getvardims(climo&lev))
> filevardef(fout, "biglev" ,typeof(bigclimo&biglev),getvardims(bigclimo&biglev))
> filevardef(fout, "lat" ,typeof(climo&lat),getvardims(climo&lat))
> filevardef(fout, "lon" ,typeof(climo&lon),getvardims(climo&lon))
> filevardef(fout, "mean",typeof(mean),getvardims(mean))
> filevardef(fout, "bigmean",typeof(bigmean),getvardims(bigmean))
> filevardef(fout, "sddev",typeof(climo),getvardims(climo))
> filevardef(fout, "bigsddev",typeof(bigclimo),getvardims(bigclimo))
>
> filevarattdef(fout,"time" ,climo&time)
> filevarattdef(fout,"lev" ,climo&lev)
> filevarattdef(fout,"biglev",bigclimo&biglev)
> filevarattdef(fout,"lat" ,climo&lat)
> filevarattdef(fout,"lon" ,climo&lon)
> filevarattdef(fout,"mean",mean)
> filevarattdef(fout,"bigmean",bigmean)
> filevarattdef(fout,"sddev",climo)
> filevarattdef(fout,"bigsddev",bigclimo)
>
> ;setfileoption(fout,"DefineMode",False)
> print("Predefining done. Writing now.")
>
> fout->sign = (/climo&sign/)
> fout->time = (/climo&time/)
> fout->lev = (/climo&lev/)
> fout->biglev = (/bigclimo&biglev/)
> fout->lat = (/climo&lat/)
> fout->lon = (/climo&lon/)
> fout->mean = (/mean/)
> fout->bigmean = (/bigmean/)
> fout->sddev = (/climo/)
> fout->bigsddev = (/bigclimo/)
>
>
> ----------------------------------------
> Kyle S. Griffin
> Department of Atmospheric and Oceanic Sciences
> University of Wisconsin - Madison
> Room 1421
> 1225 W Dayton St, Madison, WI 53706
> Email: ksgriffin2@wisc.edu
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Fri Feb 14 09:28:24 2014