NCL Home > Documentation > Functions > File IO

setfileoption

Sets a number of file-format-specific options.

Prototype

	procedure setfileoption (
		format_or_file [1] : string or file,  
		option         [1] : string,          
		value                                 
	)

Arguments

format_or_file

Either a string specifying a file format, or a reference to a file opened with addfile or addfiles. If set as a string the value must be equal to one of the suffixes accepted by NCL as the indicator of a supported format (e.g. "nc", "grb", or "GRIB"), or else "bin" to specify an option supported for binary files.

option

A case-insensitive string containing the name of an NCL-supported file option.

value

A scalar value or one-dimensional array containing the desired value(s) to be assigned to the option. The type of this argument is dependent on the requirements of the option being set.

Description

This procedure allows the user to set a number of options that are dependent on the format of the file accessed. The options have varying restrictions concerning when they may be set and the values that are acceptable. If the first argument is a string specifying a format, then the option setting applies to any file of the specified format when it is first opened using addfile or addfiles. If the first argument is the string "bin", then the option setting applies to subsequent calls to the built-in binary data reading functions or binary data writing procedures. Files referenced by existing file variables will not be affected. On the other hand, options set using a file variable as the first argument affect only the specified file, and may alter NCL's treatment of it dynamically.

NetCDF file format options

DefineMode

Note: in version 4.3.1, this option will default to True.

This option may be set only for files opened for writing. It is logical-valued with a default value of False. If set True, rather than performing each operation atomically, NCL modifies its standard behavior and holds the NetCDF library in define mode without closing the file while dimensions, attributes, and variables are defined using the functions filedimdef, fileattdef, filevardef, and filevarattdef. The file remains open in define mode until variable data is read or written, the option is set False again, or the file variable is deleted. Setting this option True when the file is first opened can improve performance dramatically for files with many variables and/or attributes, assuming that care is taken to define all dimensions, attributes, and variables and assign attribute values before writing actual data to the file.

CompressionLevel

Available in version 4.3.1 or later.

Specify the level of data compression as an integer in the range 0 through 9. Increasing values indicate greater compression. Compression is lossless. There are tradeoffs between the time spent compressing the file, versus the amount of compression achieved. Informal tests show that compression level 9 results in a file only a few percent smaller than a compression level 5 file, but it requires 4 or 5 times the amount of time to create it. (This option is ignored unless the "Format" option is set to "NetCDF4Classic" (*).)

Format

You may set this option only prior to opening a new NetCDF file in "create" mode. The first argument must be a string specifying the NetCDF format, such as "nc". This string-valued option currently has four valid values, two of which are synonyms.

The default value, "Classic", indicates that a standard NetCDF file should be created. Standard NetCDF files are more limited with respect to size. Assuming the underlying file system has support for large files, the total size can exceed 2 GB, but there are severe restrictions regarding the number of large variables and the order in which they are written. In general, the standard format should be used if and only if it is known that the total file size will be less than 2 GB.

The two synonymous values, "LargeFile" and "64BitOffset", indicate that a NetCDF file with support for larger variables and a theoretically much larger total size (about 9.22e+18 bytes) should be created. Each fixed-size variable, or each 'record' (first dimension) of a variable with an unlimited dimension of a 64-bit offset file can have a size of up to 4 GB.

Again, assuming the underlying file system has support for large files, recent versions of NCL can transparently read NetCDF files in either the classic or the 64-bit offset format. For more detailed information about large file support in NetCDF see http://www.unidata.ucar.edu/software/netcdf/docs/netcdf/Large-File-Support.html.

In version 4.3.1 or later, you can specify "NetCDF4Classic" to create a file using the NetCDF 4 classic model format. (*) The classic model constrains the interface to the constructs provided by NetCDF 3 and earlier. However, the underlying file format, like that of all NetCDF 4 files, is HDF 5. Files written in this format can take advantage of the built-in file compression available in HDF 5. Use the CompressionLevel option (see above) to enable compression. Also the HDF 5 format removes virtually all restrictions on file and individual variable size. NCL version 4.3.1 provides beta-level support for this format because NetCDF 4 and the release of HDF 5 that it depends on are both still in the beta-testing phase of development. It should probably not be used for mission-critical file creation and it is not yet available on every system that NCL runs on.

(*) Support for NetCDF 4 files is not available on every system, as NetCDF 4 is still in beta testing, and does not work for 64-bit systems.

HeaderReserveSpace

This option may be set only for files opened for writing. This integer-valued option may be used to reserve extra space at the beginning of a NetCDF file in order to define new dimensions, variables, or attributes after variable data has been written. It specifies the number of bytes to reserve in the header portion of the file in addition to the bytes used for the currently defined dimensions, variables, and attributes. Notes that the space is reserved only when variable data is actually written to the file.

MissingToFillValue

Available in version 4.2.0.a034 or later.

This logical-valued option has an effect only when set prior to opening an existing NetCDF file in read or write mode. If set to its default value, True, it causes a "virtual" _FillValue attribute to be created for any variable that has the attribute missing_value but not _FillValue. The purpose is to more gracefully handle files that use the COARDS-compliant missing_value instead of _FillValue to indicate missing data. Note that if a variable in a file has both a missing_value and a _FillValue, or if it has neither, the option does nothing. The virtual _FillValue attribute is not actually part of the NetCDF file, but only appears to be from within the NCL environment. However, it is propagated upon assignment of the variable just like any other attribute. If the file is opened for writing and you assign to the attribute, it becomes an actual attribute. Therefore to make it a permanent part of a file you could simply execute a statement such as:

f->v@_FillValue = f->v@_FillValue

Unfortunately, there is no simple way within NCL to tell whether a particular instance of a _FillValue attribute (where there is also a missing_value attribute with the same value) is virtual or actual. However note that executing the statement above is a harmless no-op if the variable's _FillValue attribute actually exists in the file.

PreFill

This option may be set only for files opened for writing. It is logical-valued with a default value of True. If set False, NCL alters the standard behavior of the NetCDF library such that variable element locations in the file are not "pre-filled" with the missing (fill) value associated with the variable. This can noticeably improve performance when writing large datasets. However, if you set this option False, you are responsible for ensuring that all the elements of the variables you have defined are assigned a valid value.

SuppressClose

Note: in version 4.3.1 and later, this option will default to True. If you need to open lots of files, however, you may want to set this option to False. The limit depends on how many file descriptors your system will allow to have open at any time.

This logical-valued option may be set for any NetCDF file. Its default value is False. Normally NCL opens a file each time an operation involving reading or writing data occurs and closes it again when the operation is complete. This helps ensure the file's integrity but may result in loss of performance. If this option is set True, NCL will only only close the file when the user invokes 'delete' on the file object variable or the option is set False again. The effect on performance of this option is not fully explored, and if NCL quits unexpectedly it is possible that a file open for writing may corrupted. Note that the DefineMode option also keeps the file open, but only until the first operation that requires leaving define mode is completed. SuppressClose and DefineMode may be used together in files open for writing.

GRIB file format options

DefaultNCEPPTable

Available in version 4.2.0.a034 or later.

This option applies only to GRIB1-formatted files.

This string-valued option has two valid values: "Operational", the default, or "Reanalysis". It specifies whether to default to the use of the NCEP operational parameter table (http://www.ncl.ucar.edu/Document/Manuals/Ref_Manual/ncep_opn.htm) or the NCEP reanalysis parameter table (http://www.ncl.ucar.edu/Document/Manuals/Ref_Manual/ncep_reanal.htm). The option only applies in cases where NCL, on its own, cannot definitively determine which of these tables to use because of historical ambiguities in NCEP usage.

InitialTimeCoordinateType

This option applies to GRIB1- and GRIB2-formatted files.

This string-valued option has two valid values: "Numeric", the default, or "String". Note that up to version 4.2.0.a034 of NCL, the default was "String". By making the default "Numeric", the initial time coordinates conform to a COARDS-compliant numeric time in hours since 1800-01-01 (00.00).

Prior to version 4.2.0.a034, NCL used an array of strings to represent the coordinate values of an initial time dimension. This representation proved to be problematic, both because it did not conform to standard CF and COARDS conventions, and because it was difficult to use in a context where numerical properties such as monotonicity were usually expected. The "String" default was originally retained for backwards compatibility, but because of the problems described above, it was decided to make the default "Numeric".

By setting this option to "String", you can maintain backwards compatibility with versions before 4.3.0. If this option is set after adding a GRIB file, any initial time dimensions in the file will immediately be updated to reflect the new option value.

Note: in NCL's representation of GRIB1 and GRIB2 files, the initial time dimension is distinguished from the forecast time dimension, whose coordinate values are numerical offsets from a particular initial time.

SingleElementDimensions

Available in version 4.3.0 or later.

This option applies to GRIB1- and GRIB2-formatted files.

This option allows the user to specify that variables with only a single initial time, forecast time, level, ensemble or probability value, usually handled as attributes, be treated as containing single element dimensions. It is a string-valued option whose default value "None" means that no single-element dimensions will appear in NCL's representation of the GRIB file. Conversely, if the option is given the value "All", then all possible dimensions will be created for each variable. Otherwise, the desired single element dimensions may be specified individually. The valid choices are "Initial_time", "Forecast_time", "Level", "Ensemble", and "Probability". Since this option can take an array of values, you can specify more than one of these dimension types as the option value. However, the options "None" and "All" override the individual dimension specifications.

Note that dimensions are not created if the variable does not have an actual value associated with the dimension type, regardless of the value given to this option. For example, variables that are not part of an ensemble forecast will never have an ensemble dimension, and variables whose level type (e.g. Tropopause) does not have a numerical value will never have a level dimension. In the case of level types, it may depend on who wrote the record: files written by some centers may give no value for certain level types where others may use the numerical value 0. The intent of this option is to make it easier to concatenate conforming variables from multiple files together.

ThinnedGridInterpolation

This option applies to GRIB1- and GRIB2-formatted files.

This string-valued option has two valid values: "Cubic" or "Linear". This option may be set for any GRIB file, but only has an effect when the GRIB file contains data on a thinned grid. Note, up to version 4.2.0.a034 of NCL, the default was "Linear". However, the "Cubic" value has been shown to produce better results, hence, as of version 4.3.0, this will become the default. The exception is when bitmask arrays are encountered. In this case, "Linear" will be the default because the "Cubic" interpolation is not applicable to bitmask arrays.

The GRIB documentation refers to these grids as "quasi-regular". The option controls the interpolation performed in converting variable data on the grid to the standard rectangular form that is returned by NCL. If this option is modified after adding a GRIB file, the values of variables defined on one of these thinned grids will be modified.

Binary data file options

ReadByteOrder

This string-valued option has three valid values: "Native", the default, "BigEndian", and "LittleEndian". It allows you to read binary data files written with either big or little endian byte ordering independent of the system on which you are running. It affects the contents of variables read using the binary data reading functions cbinread, fbindirread, fbinrecread, and fbinread, as well as the results of the record counting function fbinnumrec. When this option has its default value, "Native", these functions behave as they always have: they expect the data bytes to be ordered according to the conventions of the system on which the code is currently executing. However, if the option is set to "BigEndian" or "LittleEndian" NCL will expect the input data file to be byte-ordered according to the conventions of, respectively, a big-endian or a little-endian system. The functions will then either byte-swap the data or not, as appropriate for the byte ordering of the system executing the code, before assigning the the data to the return variable.

WriteByteOrder

This string-valued option has three valid values: "Native", the default, "BigEndian", and "LittleEndian". It allows you to write binary data files with whatever byte ordering you prefer independent of the system on which you are running. It affects the contents of files written using the binary data writing procedures cbinwrite, fbindirwrite, fbinrecwrite, and fbinwrite. When this option has its default value, "Native", these procedures behave as they always have: they write the data bytes ordered according to the conventions of the system on which the code is executing. However, if the option is set to "BigEndian" or "LittleEndian" NCL will write output data files byte-ordered according to the conventions of, respectively, a big-endian or a little-endian system. The procedure will either byte-swap the data or not, as appropriate for the byte ordering of the system executing the code, in order to produce data files with the specified byte ordering.

See Also

addfile, addfiles, filedimdef, fileattdef, filevardef, filevarattdef, cbinread, fbindirread, fbinrecread, fbinnumrec, fbinread, cbinwrite, fbindirwrite, fbinrecwrite, fbinwrite.

Examples

Example 1

Set all subsequently opened GRIB files to use cubic interpolation whenever data on a thinned (quasi-regular) grid is encountered:

   setfileoption("grb","ThinnedGridInterpolation","cubic")

Example 2

Open a NetCDF file in create mode and set DefineMode True. Define some dimensions and a variable and some attributes, then turn DefineMode off.

   f = addfile("test.nc","c")
   setfileoption(f,"DefineMode",True)
   filedimdef(f,(/"lat","lon"/), (/100,150/), (/False,False/))
   filevardef(f,"PSfc","float",(/"lat","lon"/))
   atts = 1.0
   atts@_FillValue = -999.0
   atts@long_name = "Surface Pressure"
   filevarattdef(f,"PSfc",atts)
   setfileoption(f,"DefineMode",False) ; (not really necessary)
   ; assign data only to file variable ; would automatically suspend define mode 
   f->Psfc = (/ datavar(0,:,:) /)

Example 3

Set up the binary data reading functions to expect files written with little-endian byte ordering, and the binary data writing procedures to produce big-endian files, regardless of the system on which NCL is running. Produce a big-endian file from a little-endian file containing float data.

   setfileoption("bin","ReadByteOrder","LittleEndian")
   setfileoption("bin","WriteByteOrder","BigEndian")

   v = cbinread("data.littleEndian.bin",-1,"float")
   cbinwrite("data.bigEndian.bin",v)