NCL: Write netCDF with one unlimited dimension, plus chunking and compression

Support for NetCDF-4 files is in beta-testing in V6.1.1.
We encourage people to test it out and give us feedback on our ncl-talk email list.

Below is a script which creates a variable with one unlimited dimension, and then expands it in the dimension, plus chunking and compression.

Pre-define some properties before writing the file

; This script writes a large variable to a file in NetCDF4
; with chunking and compression.

begin

 theflnm = "./nc4_chunkingNcompression.nc"
 setfileoption("nc","format","netcdf4")
 setfileoption("nc","headerReserveSpace",64000)
 setfileoption("nc","preFill",False)
;setfileoption("nc","defineMode",True)

;Remove the file if it already existed.
 if (isfilepresent(theflnm)) then
        system("rm " + theflnm)
 end if

Create the file:


 f = addfile(theflnm,"c")

Define dimensions:

(Note ONLY time dimension is unlimited, and its current value is 1.)

 ctim = 1
 ntim = 250 ;1000 will be little bit too much
 nlat = 1800
 nlon = 3600
 dim_names = (/"time","lat","lon"/)
;time dimension is unlimited, but currently set to 1.
 dim_sizes = (/ctim,nlat,nlon/)
 dim_unlimited = (/True,False,False/)

 print("Defining dimension sizes to be " + dim_sizes)
 filedimdef(f,dim_names,dim_sizes,dim_unlimited)

Define the chunk sizes:

 mtim = 1
 mlat = nlat/10
 mlon = nlon/10
 chunk_sizes = (/mtim,mlat,mlon/)
 filechunkdimdef(f,dim_names,chunk_sizes,dim_unlimited)

Define some file properties:

 atts = True
 atts@description = "This file tests NCL's ability to create >2GB variables"
 atts@timestamp = systemfunc("date")
 print("Defining file attributes")
 fileattdef(f,atts)

Define file variables:

 print("Defining file variables")
 filevardef(f,"time","float","time")
 filevardef(f,"lat","float","lat")
 filevardef(f,"lon","float","lon")
 filevardef(f,"wave","float",(/"time","lat","lon"/))

Define file variables attributes:

 print("Defining file variable attributes")
 tatts = True
 tatts@long_name = "elapsed time"
 tatts@units = "hours since 01-01-2000"
 filevarattdef(f,"time",tatts)

 latatts = True
 latatts@long_name = "latitude"
 latatts@units = "degrees_north"
 filevarattdef(f,"lat",latatts)

 lonatts = True
 lonatts@long_name = "longitude"
 lonatts@units = "degrees_east"
 filevarattdef(f,"lon",lonatts)

 waveatts = 0.0
 waveatts@long_name = "Sine Wave on the sphere"
 waveatts@units = "None"
 waveatts@_FillValue = 1e32
 filevarattdef(f,"wave",waveatts)

 filevarchunkdef(f, "wave", chunk_sizes)
 filevarcompressleveldef(f, "wave", 2)

Create coordinate arrays:

(Note lon and lat write out here, but time will write later.)

 print("Creating coordinate arrays")
 time = fspan(0,ntim-1,ntim)
 lat = fspan(-90,90,nlat)
 f->lat = (/lat/)
 lon = fspan(-180,180,nlon)
 f->lon = (/lon/)

Generate a 2D array (and make it not too boring).

 print("Generating " + nlat + " x " + nlon + " array")
 base = new((/nlat, nlon/), "float", waveatts@_FillValue)
 wave = new((/nlat, nlon/), "float", waveatts@_FillValue)

 arc = 2.0 * 3.141693 / 180.0

 lon = sin(arc * lon)
 lat = cos(arc * lat)

 do j = 0, nlat -1
    base(j, :) = lat(j) * lon
 end do

Loop through time, and write out the data (like integrating a model).

 print("Looping across time (" + ntim + ")")
 do n = 0, ntim-1
    tf = sin(0.01 * arc * (n + 1))
   ;generate a 2d array
    b1 = base(:, nlon - 1)
    base(:, 1:nlon - 1) = base(:, 0:nlon - 2)
    base(:, 0) = b1;

    wave = tf * base;

    f->time(n) = (/time(n)/)
    f->wave(n,:,:) = (/wave/)
    command = "date; ls -l " + theflnm
    m = n + 1
    print(systemfunc(command) + " : timestep " + m + "/" + ntim)
 end do

The end.

 print("Done looping across time (" + ntim + ")")

 delete(f)
end

This script produced (use "ncl_filedump nc4_chunkingNcompression.nc"):

Variable: f
Type: file
filename:       nc4_chunkingNcompression
path:   nc4_chunkingNcompression.nc
attributes:
    timestamp   :       Fri Jan 25 11:33:34 MST 2013
    description :       This file tests NCL's ability to create >2GB variables

dimensions:
    time        = 250 // unlimited
    lat = 1800
    lon = 3600

chunk dimensions:
    time        = 1 // unlimited
    lat = 180
    lon = 360

variables:
    time:        [ 250  ]
        Chunking Info:  [ 1  ]
        attributes:
            units       :       hours since 01-01-2000
            long_name   :       elapsed time

    lat:         [ 1800  ]
        Chunking Info:  [ 1  ]
        attributes:
            units       :       degrees_north
            long_name   :       latitude

    lon:         [ 3600  ]
        Chunking Info:  [ 1  ]
        attributes:
            units       :       degrees_east
            long_name   :       longitude

    wave:        [ 250 , 1800 , 3600  ]
        Chunking Info:  [ 1 , 180 , 360  ]
        attributes:
            _FillValue  :       100000003318135351409612647563264.000000
            units       :       None
            long_name   :       Sine Wave on the sphere