Usage help page
This page provides helpful hints for using NCL efficiently and effectively. This is not a complete and comprehensive usage FAQ, but it lists some very important issues that users should be aware of when using NCL. A general working knowledge of the NCL syntax is required for this page.
- Writing efficient NCL source
- Remove loops by using NCL Array Syntax
- Remove scalar expressions from loops
- Avoid reordering arrays if possible
- Use built-in functions and procedures when applicable
- Know when memory is allocated
- Learn and take advantage of all three styles of NCL subscripting to subset variables
- Reduce disk I/O when working with files
- Build custom functions in Fortran or C when NCL is too slow
- File I/O issues
- Differences between various supported formats with respect to efficiency
- Writing NetCDF files efficiently
- Checking for correct syntax of scripts before execution
- Using command line options
Writing efficient NCL source
There are many features in NCL that when not used -- or used improperly -- result in inefficient data processing. Many of these problems result from the fact that NCL is an interpreted language and must do additional processing for each statement, subscript, and expression executed. Therefore a good rule of thumb is to strive to reduce the number of statements, subscripts, and expressions executed.
The most important fundamental to remember is that NCL is more efficient when NCL's array operations are used. Array operations mean an entire array is used rather than a subscripted element of the array. Consider multiplying two 2-dimensional floating point arrays dimensioned 100x100. One way would be to loop from 0-99 for both dimensions and subscripting individual elements of each array, multiplying them together and assigning them to a result.
do i = 0,99 do j = 0,99 c(i,j) = a(i,j)*b(i,j) end do end do
Although intuitive for most programmers, this is the least efficient way to do this in NCL. All of NCL's algebraic operators allow arrays of similar dimensions and types to be used as operand. For example, the above loop could be rewritten as follows:
c = a*bBy counting each each statement, subscript, and expression in the original loop, you would get three subscripts, one statement, and one expression. These values must be multiplied by 10000 since there are that many iterations, bringing the totals to 30000 subscripts, 10000 statements, and 10000 expressions. The second example has one statement and one expression. By removing unnecessary loops and using NCL's array syntax in NCL source, very large performance gains can be made.
Removing scalar expressions
Sometimes it is impossible to completely remove loops. When this is the case, attention should be focused on removing scalar expressions or unnecessary expressions and statements inside of loops. Consider the following example:
do i = 0,999 do j = 0,999 T(i,j) = 100.0 - 8 * sqrt(i^2 + j^2) end do end do
Using the counting scheme outlined above, this loop has one subscript, one statement, and six expressions (the function sqrt is counted as an expression). These must be multiplied by 1000000 iterations. Now, using a little algebra and knowing that the operators and functions can accept entire arrays as well as individual elements, the above loops can be rewritten as:
do i = 0,999 do j = 0,999 T(i,j) = i^2 + j^2 end do end do T = 100 - 8 * sqrt(T)This version of the loops contains one subscript, one statement, and three expressions times 1000000 iterations. The additional line adds merely three total expressions, making the total just slightly over half the original. In this example, operations common to all elements of the array T were moved outside the loop. By the way, the above loops are not actually necessary. They can be completely removed by initializing T using ispan. This is left as an exercise for the reader.
Avoid reordering arrays if possibleReordering arrays can be an expensive memory and time operation, because a full copy of the array is made in order to reorder it.
If you are reordering the same array multiple times, then it might be faster to do this just once and save it to a new variable:
t_reorder = t(lat|:,lon|:,time|:)
If you are reordering an array in order to call a function xxxx that requires the dimensions in a particular order, then check if this function has a xxxx_n equivalent.
For example, if you have z(time,lat,lon) and want an average across time:
zAvgTime = dim_avg( z(lat|:, lon|:, time|:) ) ; ==> zAvgTime(nlat,nlon)
then this is a case where dim_avg_n would be useful:
zAvgTime = dim_avg_n( z, 0 ) ; 0 = time dimension
Many of the dim_xxxx_n functions have dim_xxxx_n_Wrap equivalents. If metadata is desired, use the "_Wrap" version of the function:
zAvgTime = dim_avg_n_Wrap( z, 0 )
Use built-in functions and procedures when applicableNCL has many time-saving functions and procedures that operate on entire arrays. Before using loops to compute values, check the built-in function set to see if something is available. The following are some of the most useful time-saving functions.
- ndtooned and onedtond
- max, min, avg, sum, product, stddev, variance
- dimsizes, filevardimsizes
Know when memory is allocatedMemory allocation in NCL occurs during a new call, assignment, subscripting, and expression evaluation. In the cases of subscripting and expression evaluation, this memory is temporary and NCL will either free it or reuse it as efficiently as possible.
Avoid using new to create variables unless it is absolutely necessary. Consider the following source:
T = new(filevardimsizes(file1,"T"),float) . . . T = file1->T
This causes an extra array the size of variable T to be unnecessarily allocated. The file variable reference, file1->T, needs to allocate an array the size of T in order to read it from the file. At this point there are two arrays. During the assignment, one array is copied to the other and then freed. This is unnecessary since NCL will implicitly define variables when they appear on the left side of an assignment. Therefore the above statements should be written as:
T = file1->T
This way the temporary value allocated by the file read is merely assigned to the variable T. In this way only one allocation occurs. One example of when this is unavoidable is when reading from multiple files into one array. The following is an example of that:
filenames = (/"a.nc","b.nc","c.nc","e.nc"/) file1 = addfile(filenames(0),"r") dims = filevardimsizes(file1,"T") T = new((/dimsizes(filenames),dims(0),dims(1),dims(2)/),float) T(0,:,:,:) = file1->T do i = 1,dimsizes(filenames)-1 file1 = addfile(filenames(i),"r") T(i,:,:,:) = file1->T end do
Learn and take advantage of all three styles of NCL subscripting to subset variables
NCL has a very powerful variable subscripting syntax with features not found in other languages. Learning when to use these features can be critical for writing efficient NCL source.
Consider the operation of transposing a two-dimensional array. One could write the following inefficient source;
dims = dimsizes(T) Ttranspose = new((/dims(1),dims(0)/),float) do i = 0, dims(1)-1 do j= 0, dims(0)-1 Ttranspose(i,j) = T(j,i) end do end doAfter taking the time to learn learn NCL Named Subscripting, the above can be rewritten in just three lines of source:
T!0 = "x" T!1 = "y" Ttranspose = T( y | :, x | :)
Consider the operation of reversing the order of a dimension:
dims = dimsizes(T) do i = 0,dims(1)/2 -1 tmp = T(:,i) T(:,i) = T(:,(dims(1)-1) - i) T(:,(dims(1)-1) - i) = tmp end doThe above operation can be rewritten using only one line of NCL source:
T = T(:,::-1)It pays to take the time to learn NCL's unique syntax.
Reduce disk I/O when working with files
Disk I/O is usually much slower than memory accesses, therefore it is recommended that searches or access to individual elements of arrays in files be reduced to improve performance. For example, the first script will run much faster than the second because the search variable is not read from disk on every iteration of the loop. The following file "sao.cdf" contains a two-dimensional character array called "id" in which the three-character station ids are stored. The loops look for the index of a specific id.Faster loop:
file1 = addfile("sao.cdf","r") id = file1->id do i = 0, dimsizes(id(:,0)) - 1 if( id(i,:) .eq. "DEN") break end if end do print(i)Slower loop:
file1 = addfile("sao.cdf","r") do i = 0, dimsizes(file1->id(:,0)) - 1 if( file1->id(i,:) .eq. "DEN") break end if end do print(i)
It can be time-consuming to reorder dimensions when reading a variable off a file. If you can spare the extra memory, it is better to read the variable into memory, and then reorder the dimensions: Faster code:
dims = getfilevardims(file1,"T") T = file1->T Tnew = T( $dims(1)$ | :, $dims(0)$ | :, $dims(2)$ | :)Slower code:
dims = getfilevardims(file1,"T") Tnew = file1->T( $dims(1)$ | :, $dims(0)$ | :, $dims(2)$ | :)
Build custom functions in Fortran or C when NCL is too slowNCL has the ability to load dynamic shared libraries which can contain Fortran or C functions and procedures. Using external Fortran and C functions is highly recommended when trying to boost NCL's performance. To configure external functions and procedures, a C source code wrapper must be written. This wrapper de-references parameters from NCL's stack and passes them to the C or Fortran routine. A utility called wrapit77 is available to automatically generate wrappers for Fortran source code. All the user has to do is place the comment "C NCLFORTSTART" before the subroutine or function declaration and "C NCLEND" after the parameter data declarations. Then the user executes the command wrapit77 < file.f > file_W.c to generate the C source code wrapper. Next they compile and link into a shared library the Fortran and C files. The shared object is loaded using the external command. For more information on this see Extending the NCL function and procedure set.
File I/O issues
Each supported data format may have its own limitations with respect to operations provided by NCL. See the Supported data format information for specific information, conventions, and limitations on each data format in NCL. The following passages are recommendations for using files and working with them.
If you don't know the names of the variables in a file, use the function getfilevarnames to achieve this. The syntax for using a string to read a variable is:
names = getfilevarnames(file1) var0 = file1->$names(i)$
The '$' operator tells NCL to use the string values between the '$'s as the name of the variable to read. Similarly, attributes and coordinate variables can be read in this fashion.
It is often necessary to know the dimension sizes of variables in files. Never use the dimsizes function for this. This will cause the entire variable to be read in and then discarded. Use the filevardimsizes function for this.
Differences between various supported formats with respect to efficiencyIt is important to understand efficiency issues with respect to the different formats supported by NCL. There are big differences between the NetCDF, GRIB, and CCM formats. Both CCM and GRIB are not by definition self-describing. This means the entire files must be scanned from beginning to end when opened with the addfile command. Because of this, there is a noticeable overhead to opening these files when they are large. Knowing this can save time when designing scripts. By contrast, the expense of opening HDF and NetCDF files is almost negligible
Writing NetCDF files efficiently
There are some built-in procedures available that help write files efficiently. NetCDF has some strange performance problems stemming from how the data are arranged on disk. Attributes and ancillary information, if written after data, can cause a tremendous amount of file copies which slow NCL down.
By pre-defining the dimensions, attributes, and variables, much time can be saved. Essentially the rule of thumb is to define and write the data in the order it will be laid out in the file. The four procedures are filedimdef, filevardef, filevarattdef, fileattdef. The basic strategy is to first define all the dimensions and then all the variables in the order they'll be written. If at all possible, define one dimension as unlimited. The unlimited dimension initially has no size, meaning any variables added to the file containing the unlimited dimension will also have no size. Variables with no size can be added without incurring a file copy. After adding each variable, add its attributes. By proceeding in this fashion, the file copies are minimized, and when eventual assignment to the variables occurs, file copies again are minimized.
To further speed things up, use the DefineMode setting in the setfileoption procedure, along with the above-mentioned procedures. This has been shown in one case to speed up the writing of a NetCDF file (which had lots of variables), from 8 minutes to 8 seconds.
Using command line optionsYou can use NCL's command line option capability to set NCL variables on the UNIX command line. This enables you to easily set NCL variables on the UNIX command line when you run NCL.
Defining functions and procedures in scripts
Because functions and procedures can only be defined once, it is important not to put them in scripts that are intended to be run multiple times from the same invocation of NCL. As of release 4.1, functions and procedures can be undefined using the undef procedure. When writing functions and procedures that will be loaded, it is probably a good idea to insert an undef call immediately before the definition.
undef("mymax") function mymax(a,b) begin . . .
Use of blocks
When writing scripts, it can be very useful to enclose the script in a block. This forces NCL to scan in the entire source of the script before executing any of the commands nested in the block. This means any syntax errors will be reported before commands are executed. For example, by placing a begin and end around the following, the syntax error after the long loop is detected before the loop is executed.
begin tmp = new((/1000000/),float) do i = 0,999999 tmp(i) = i end do asciiwrite("tmp.ascii",tmp(500000:) end