Re: Issue with writing large data out to netCDF

Someone more knowledgeable than me will have to answer
with a definitive answer. See:

      http://www.ncl.ucar.edu/FAQ/#file_io_004

---
  Note the end of the above FAQ:
"... the current recommendation from Unidata is to maintain
backwards compatibility where possible for apps that can only
deal with the original small file format of NetCDF."
NCL adheres to this recommendation.
The original "small file format of NetCDF" is netCF-3
So if you are writing multiple variables You should always use:
   setfileoption("nc","Format","LargeFile")
or
   setfileoption("nc","Format","NetCDF4Classic")
On 2/14/14, 8:08 AM, Kyle Griffin wrote:
> Hi all,
> I've recently been having some issues writing out large variables (>2GB) to
> a netCDF file and I'm not sure why these limits are present. Via some
> testing, I was able to isolate that the problem does seem to occur when the
> individual variables cross that magical 2 GB threshold. This code is
> essentially identical to that which has been used on a number of other
> occasions to write large (>10GB) variables to netCDF, but with only one
> variable in a file. I was also defining "netcdf4classic" in my previous
> writing-large-variable experience, but I don't believe this should make a
> difference and the example below is almost entirely in-line with the
> example on writing large variables.
>
> Any advice on where my code is going wrong? A bit more info on the sizes of
> the variables is below. Thanks for any help.
>
>
> Kyle
>
> Output:
>
> fatal:NetCDF: Operation not allowed in define mode: error attempting to
> write variable (sign) to file
> (/glade/scratch/kgriffin/dp/TEST_dp_stddev_1201.nc)
> fatal:["Execute.c":8128]:Execute: Error occurred at or near line 234 in
> file ./two-way-climo.ncl
> ;.....code continues past this, and then dies at:...........
> ncendef: ncid 131072: NetCDF: One or more variable sizes violate format
> constraints
> ncclose: ncid 131072: NetCDF: One or more variable sizes violate format
> constraints
>
>
> Variables:
>
> sign, time, lev, biglev, lat, lon are all 1-D coordinate variables.
> mean and sddev are of size [2 x 124 x 16 x 361 x 720] and type float -
> roughly 3.8GB.
> bigmean and bigsddev are the same size, but with only 3 instead of 16 for
> the third dimension.
>
>
> File-writing code:
>
>    system("rm -f "+outfile)
>    fout  = addfile (outfile, "c")
>    setfileoption(fout,"format","netcdf4")
>    ;setfileoption(fout,"DefineMode",True)
>    fAtt               = True
>    fAtt@creation_date = systemfunc ("date")
>    fileattdef( fout, fAtt )
>
>    dimNames = (/"sign","time","lev","biglev","lat", "lon"/)
>    dimSizes =
> (/dimsizes(climo&sign),dimsizes(climo&time),dimsizes(climo&lev),dimsizes(bigclimo&biglev),dimsizes(climo&lat),
> dimsizes(climo&lon) /)
>    dimUnlim = (/False,False,False,False,False, False/)
>    filedimdef(fout,dimNames,dimSizes,dimUnlim)
>
>    filevardef(fout, "sign"  ,typeof(climo&sign),getvardims(climo&sign))
>    filevardef(fout, "time"  ,typeof(climo&time),getvardims(climo&time))
>    filevardef(fout, "lev"  ,typeof(climo&lev),getvardims(climo&lev))
>    filevardef(fout, "biglev"
>   ,typeof(bigclimo&biglev),getvardims(bigclimo&biglev))
>    filevardef(fout, "lat"  ,typeof(climo&lat),getvardims(climo&lat))
>    filevardef(fout, "lon"  ,typeof(climo&lon),getvardims(climo&lon))
>    filevardef(fout, "mean",typeof(mean),getvardims(mean))
>    filevardef(fout, "bigmean",typeof(bigmean),getvardims(bigmean))
>    filevardef(fout, "sddev",typeof(climo),getvardims(climo))
>    filevardef(fout, "bigsddev",typeof(bigclimo),getvardims(bigclimo))
>
>    filevarattdef(fout,"time"  ,climo&time)
>    filevarattdef(fout,"lev"  ,climo&lev)
>    filevarattdef(fout,"biglev",bigclimo&biglev)
>    filevarattdef(fout,"lat"  ,climo&lat)
>    filevarattdef(fout,"lon"  ,climo&lon)
>    filevarattdef(fout,"mean",mean)
>    filevarattdef(fout,"bigmean",bigmean)
>    filevarattdef(fout,"sddev",climo)
>    filevarattdef(fout,"bigsddev",bigclimo)
>
>    ;setfileoption(fout,"DefineMode",False)
>    print("Predefining done. Writing now.")
>
>    fout->sign   = (/climo&sign/)
>    fout->time   = (/climo&time/)
>    fout->lev    = (/climo&lev/)
>    fout->biglev = (/bigclimo&biglev/)
>    fout->lat    = (/climo&lat/)
>    fout->lon    = (/climo&lon/)
>    fout->mean   = (/mean/)
>    fout->bigmean = (/bigmean/)
>    fout->sddev  = (/climo/)
>    fout->bigsddev = (/bigclimo/)
>
>
> ----------------------------------------
> Kyle S. Griffin
> Department of Atmospheric and Oceanic Sciences
> University of Wisconsin - Madison
> Room 1421
> 1225 W Dayton St, Madison, WI 53706
> Email: ksgriffin2@wisc.edu
>
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk