Issue with writing large data out to netCDF

Hi all,
I've recently been having some issues writing out large variables (>2GB) to
a netCDF file and I'm not sure why these limits are present. Via some
testing, I was able to isolate that the problem does seem to occur when the
individual variables cross that magical 2 GB threshold. This code is
essentially identical to that which has been used on a number of other
occasions to write large (>10GB) variables to netCDF, but with only one
variable in a file. I was also defining "netcdf4classic" in my previous
writing-large-variable experience, but I don't believe this should make a
difference and the example below is almost entirely in-line with the
example on writing large variables.

Any advice on where my code is going wrong? A bit more info on the sizes of
the variables is below. Thanks for any help.

Kyle

Output:

fatal:NetCDF: Operation not allowed in define mode: error attempting to
write variable (sign) to file
(/glade/scratch/kgriffin/dp/TEST_dp_stddev_1201.nc)
fatal:["Execute.c":8128]:Execute: Error occurred at or near line 234 in
file ./two-way-climo.ncl
;.....code continues past this, and then dies at:...........
ncendef: ncid 131072: NetCDF: One or more variable sizes violate format
constraints
ncclose: ncid 131072: NetCDF: One or more variable sizes violate format
constraints

Variables:

sign, time, lev, biglev, lat, lon are all 1-D coordinate variables.
mean and sddev are of size [2 x 124 x 16 x 361 x 720] and type float -
roughly 3.8GB.
bigmean and bigsddev are the same size, but with only 3 instead of 16 for
the third dimension.

File-writing code:

  system("rm -f "+outfile)
  fout = addfile (outfile, "c")
  setfileoption(fout,"format","netcdf4")
  ;setfileoption(fout,"DefineMode",True)
  fAtt = True
  fAtt@creation_date = systemfunc ("date")
  fileattdef( fout, fAtt )

  dimNames = (/"sign","time","lev","biglev","lat", "lon"/)
  dimSizes =
(/dimsizes(climo&sign),dimsizes(climo&time),dimsizes(climo&lev),dimsizes(bigclimo&biglev),dimsizes(climo&lat),
dimsizes(climo&lon) /)
  dimUnlim = (/False,False,False,False,False, False/)
  filedimdef(fout,dimNames,dimSizes,dimUnlim)

  filevardef(fout, "sign" ,typeof(climo&sign),getvardims(climo&sign))
  filevardef(fout, "time" ,typeof(climo&time),getvardims(climo&time))
  filevardef(fout, "lev" ,typeof(climo&lev),getvardims(climo&lev))
  filevardef(fout, "biglev"
 ,typeof(bigclimo&biglev),getvardims(bigclimo&biglev))
  filevardef(fout, "lat" ,typeof(climo&lat),getvardims(climo&lat))
  filevardef(fout, "lon" ,typeof(climo&lon),getvardims(climo&lon))
  filevardef(fout, "mean",typeof(mean),getvardims(mean))
  filevardef(fout, "bigmean",typeof(bigmean),getvardims(bigmean))
  filevardef(fout, "sddev",typeof(climo),getvardims(climo))
  filevardef(fout, "bigsddev",typeof(bigclimo),getvardims(bigclimo))

  filevarattdef(fout,"time" ,climo&time)
  filevarattdef(fout,"lev" ,climo&lev)
  filevarattdef(fout,"biglev",bigclimo&biglev)
  filevarattdef(fout,"lat" ,climo&lat)
  filevarattdef(fout,"lon" ,climo&lon)
  filevarattdef(fout,"mean",mean)
  filevarattdef(fout,"bigmean",bigmean)
  filevarattdef(fout,"sddev",climo)
  filevarattdef(fout,"bigsddev",bigclimo)

  ;setfileoption(fout,"DefineMode",False)
  print("Predefining done. Writing now.")

  fout->sign = (/climo&sign/)
  fout->time = (/climo&time/)
  fout->lev = (/climo&lev/)
  fout->biglev = (/bigclimo&biglev/)
  fout->lat = (/climo&lat/)
  fout->lon = (/climo&lon/)
  fout->mean = (/mean/)
  fout->bigmean = (/bigmean/)
  fout->sddev = (/climo/)
  fout->bigsddev = (/bigclimo/)

----------------------------------------
Kyle S. Griffin
Department of Atmospheric and Oceanic Sciences
University of Wisconsin - Madison
Room 1421
1225 W Dayton St, Madison, WI 53706
Email: ksgriffin2@wisc.edu

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Fri Feb 14 08:09:12 2014