Re: Some issues when writing variable size greater than 2 GB in ncl_6.0.0_beta

From: Mary Haley <haley_at_nyahnyahspammersnyahnyah>
Date: Thu Mar 03 2011 - 08:44:47 MST

Hi,

In order to write large (> 2 GB) variables to a NetCDF file you have to set the NetCDF "Format" option to "LargeFile" or to "NetCDF4Classic". This requires calling setfileoption. See http://www.ncl.ucar.edu/Document/Functions/Built-in/setfileoption.shtml for more info -- you can ignore the bit about the NetCDF4 classic support being "beta-level" -- that is due to be removed:

setfileoption("nc","Format","LargeFile")

or

setfileoption("nc","Format","NetCDF4Classic")

The reason why NCL doesn't just set this automatically when a user tries to write a large variable to a file is that the NetCDF library requires that this decision must be made when the file is first created, before there is any knowledge of the size of variables that will be written. It is possible that eventually we will make the "LargeFile" format the default NetCDF format, but so far the recommendation from Unidata is to maintain backwards compatibility where possible for apps that can only deal with the original small file format of NetCDF.

There's another possible solution here, but I believe this is only an issue if your variable is larger than 4 GB. It has to do with NetCDF limitations on fixed-sized variables:

http://www.unidata.ucar.edu/software/netcdf/docs/netcdf/NetCDF-64-bit-Offset-Format-Limitations.html

The important line is:

No fixed-size variable can require more than 2^32 - 4 bytes (i.e. 4GiB - 4 bytes, or 4,294,967,292 bytes) of storage for its data, unless it is the last fixed-size variable and there are no record variables. When there are no record variables, the last fixed-size variable can be any size supported by the file system, e.g. terabytes.

The fix is to make the first (leftmost) dimension "unlimited" by calling "filedimdef" before you write the variable:

     filedimdef(in2,"time",-1,True)

You can use both methods described above together.

--Mary

On Mar 3, 2011, at 7:17 AM, Sabeerali(sebi) wrote:

>
> I successfully installed the new ncl_6.0.0_beta in 64 bit Linux machines. I noted an error when we write the variable whose size is greater than 2GB into a netcdf. The error is as follows.
>
> ncendef: ncid 131072: NetCDF: One or more variable sizes violate format constraints
> ncvarput: ncid 131072: NetCDF: Operation not allowed in define mode
> fatal:NetCdf: An error occurred while attempting to write variable (time) to file (./dly.rfa.rmv3har.1978.08.nc)
> warning:FileWriteVarVar: Could not write coordinate variable (time) to file (dly.rfa.rmv3har.1978.08), continuing anyway
> ncredef: ncid 131072: NetCDF: Operation not allowed in define mode
> ncendef: ncid 131072: NetCDF: One or more variable sizes violate format constraints
> ncvarput: ncid 131072: NetCDF: Operation not allowed in define mode
> fatal:NetCdf: An error occurred while attempting to write variable (lat) to file (./dly.rfa.rmv3har.1978.08.nc)
> warning:FileWriteVarVar: Could not write coordinate variable (lat) to file (dly.rfa.rmv3har.1978.08), continuing anyway
> ncredef: ncid 131072: NetCDF: Operation not allowed in define mode
> ncendef: ncid 131072: NetCDF: One or more variable sizes violate format constraints
> ncvarput: ncid 131072: NetCDF: Operation not allowed in define mode
> fatal:NetCdf: An error occurred while attempting to write variable (lon) to file (./dly.rfa.rmv3har.1978.08.nc)
> warning:FileWriteVarVar: Could not write coordinate variable (lon) to file (dly.rfa.rmv3har.1978.08), continuing anyway
> warning:Execute: Error occurred at or near line 41 in file rfa.filt.ncl
>
>
> The script is given below.
>
>
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/diagnostics_cam.ncl"
> begin
>
> in2=addfile("cntrl_precip_1978_2008.nc","r")
>
> rf=in2->RF
> rf!0="time"
> rf!1="lat"
> rf!2="lon"
>
> time=in2->TIME1
> TIME = ut_calendar(time, 0)
> year = floattointeger( TIME(:,0) )
> month = floattointeger( TIME(:,1) )
> day = floattointeger( TIME(:,2) )
> ddd = day_of_year(year, month, day)
> yyyyddd = year*1000 + ddd
>
>
> rf_clim=clmDayTLL(rf,yyyyddd)
> smth_rf_clim=smthClmDayTLL(rf_clim,3)
> rf_anom=calcDayAnomTLL(rf,yyyyddd,smth_rf_clim)
>
> diro ="./"
> filo = "dly.rfa.rmv3har.1978.08.nc"
> system("/bin/rm -f "+diro+filo) ; rm any pre-existing file
> f = addfile(diro+filo, "c")
> f->rfa_rmv3h =rf_anom
> end
>
>
>
> The printVarSummary(rf_anom) gives the following.
>
> Variable: rf_anom
> Type: float
> Total Size: 2318745600 bytes
> 579686400 values
> Number of Dimensions: 3
> Dimensions and sizes: [time | 11322] x [lat | 160] x [lon | 320]
> Coordinates:
> time: [ 0..11322]
> lat: [-89.14151942646112..89.14151942646112]
> lon: [ 0..358.875]
> Number Of Attributes: 2
> _FillValue : -1e+34
> long_name : Anomalies: PRECIP*86400
>
>
> Note:There is no error we save the variable whose size less than 2GB.
>
> --
> **********************************
> Sabeerali C T
> Senior Research Fellow
> Climate and Global Modeling Division
> Indian Institute of Tropical Meteorology
> Pashan, Pune, 411 008
> Phone (Office): 02025904306
> ****************************************
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Thu Mar 3 08:44:59 2011

This archive was generated by hypermail 2.1.8 : Thu Mar 03 2011 - 10:00:25 MST