Better than free: Data explorations with public data and software tools

2016 SEA Conference Tutorial

Wednesday, April 6, 2016

This is a help page for students attending this one-day tutorial. The focus will be on:

  • Downloading data from NCAR's Research Data Archive (RDA)
  • Using free tools to examine, read, analyze, and visualize data

Useful links:

Useful RDA BLOG articles

Free Tools

Scripting languages
NCL Tailored for climate sciences; File I/O, computations, 2D visualizations
Python Popular programming language spanning numerous disciplines
GUI-based quick look tools
ncview Quick look at NetCDF data
Panoply Quick look at NetCDF (and other formats)
Python tools
ESMPy Python interface to ESMF regridding utility
Integrated Ocean Observing System (IOOS) Collection of software to support IOOS
Iris Data abstraction library for meteorology and climatology
matplotlib 2D plotting library
numpy Fundamental package for scientific computing with Python
pandas Data analysis tool
PyAverager Parallel tool for computing averages from climate model output
PyNGL 2D plotting library (based on NCL's graphical library)
PyNIO Read/write access to a variety of data formats (based on NCL's file I/O library)
PyReshaper Parallel tool for converting NetCDF files from time-slice (history) format to time-series (single-variable) format.
xarray Extension of pandas for multi-dimensional arrays
Command line tools
convert Manipulate graphical images for web, powerpoint, presentations, etc.
(part of ImageMagick tool suite)
NetCDF Operators (NCO) Operators to manipulate and analyze (mainly) NetCDF data
Climate Data Operators (CDO) Operators to manipulate and analyze climate and NWP model data
wgrib Manipulate, inventory and decode GRIB files
wgrib2 "Four drawers of kitchen utensils as well as the microwave and blender"
CF Compliance Checkers A perl-based CF compliance checker
Python platforms
Anaconda Data science platform for python community, gives you lots of python packages already built (NumPy, SciPy, Pandas, Jupyter, etc)
Includes "conda" package installer
miniconda "Mini" Anaconda (gives you python and conda)
Jupyter Notebook (used to be called IPython Notebook) A web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text. Works with Python, R, and other languages.

Sample NCL and Python scripts to use

Most of the datasets below were downloaded from the RDA, and we've included their identifier, "dsnnn.n".
You can search for the datasets in this group by typing the "nnn.n" identifier in the RDA's "Go to Dataset" search window.

ds604.0 -
rectilinear, 450 x 900 (lat x lon)
cfdda_plot_var.ncl plots given variable using NCL plots given variable using matplotlib plots given variable using PyNGL
ds083.2 - fnl_20070703_06_00.grib1
rectilinear, 181 x 360 (lat x lon)
fnl_grib1_plot_var.ncl plots given variable using NCL plots all variables on the file using PyNGL (needs
ds083.3 - gdas1.fnl0p25.2016030100.f00.grib2
rectilinear, 720 x 1440 (lat x lon)
gdas1_fnl_grib2_plot_var.ncl plots the given variable on the file using NCL plots all variables on the file using PyNGL (needs
ds728.3 -
rectilinear, 30 x 180 x 360 (time x lat x lon)
gpcp_plot_precip.ncl plots precipitation using NCL (global map)
gpcp_plot_precip_subset.ncl plots precipitation over the United States Uses "xarray" to compute mean across time, plots with matplotlib
Using data read off THREDDS data server Read file from OPeNDAP server and plot precipitation using matplotlib
gpcp_opendap_plot_precip.ncl Read file from OPeNDAP server (that Tom demoed) and plot precipitation using NCL
ds510.0 - gdas1.t18z.sfluxgrbf03.grib2
rectilinear, 880 x 1760 (lat x lon)
grib2_plot_var.ncl plot the given variable using NCL (needs create_html.ncl) plot all 2D variables on a GRIB2 file using matplotlib (needs plot the given variable using matplotlib plot the given variable using PyNGL
ds463.4 -
296096 time steps
hadley_timeseries_plot.ncl Creates a timeseries plot of temperature using NCL
hadley_timeseries_plot_subset.ncl Same, but with a subset of the time range
(old file, not from RDA), curvilinear, 113 x 151 (lat x lon) plot the given 2D variable using matplotlib plot the given 2D variable using PyNGL plot all 2D variables and create a simple HTML file with sample plots plot all 2D variables and create a simple HTML file with sample plots (needs vector plot using matplotlib / basemap vector plot using PyNGL
grib1_plot_uv.ncl vector plot using NCL
Sample GRIB2 (not from RDA), rotated grid, curvilinear, 461 x 421 (lat x lon) plots data in rotated projection using PyNGL plots non-rotated data using PyNGL plots non-rotated data using matplotlib
Jupyter (IPython) Notebooks
To run a notebook, type "ipython notebook" or "jupyter notebook" on the UNIX command line. A jupyter browser window should come up with a list of your *.ipynb files. Select one of these files to start working in your notebook.
fillplot_demo.ipynb A sample notebook that creates some XY plots using dummy. This is a good demo for playing with graphics inside a notebook.
pynio_demo_examine_files.ipynb A notebook for opening various files (NetCDF, GRIB) and examining them using PyNIO.
pynio_demo_query_file_contents.ipynb A notebook for opening various files (NetCDF, GRIB) and querying about the file contents.
pynio_demo_read_vars.ipynb A notebook for reading variables off various files (NetCDF, GRIB)

NCL and Python tools on yellowstone

If you have a yellowstone account:

ssh -Y
module load gnu/4.8.2
module load ncl
module load python/2.7.7
module load all-python-libs

For the latest PyNIO and PyNGL:

setenv PYTHONPATH /glade/apps/opt/PyNGL/1.5.0/gnu/4.8.2/PyNGL:/glade/apps/opt/PyNIO/1.5.0/gnu/4.8.3/PyNIO:$PYTHONPATH


export PYTHONPATH=/glade/apps/opt/PyNGL/1.5.0/gnu/4.8.2/PyNGL:/glade/apps/opt/PyNIO/1.5.0/gnu/4.8.3/PyNIO:$PYTHONPATH

Software preparation for tutorial

  • Sign up for an account on NCAR's Research Data Archive (RDA) by clicking on the "Register Now" link at the top of the page. This can take a few hours before the registration is active.

  • Install Anaconda - This will give you a scientific python environment and the "conda" command for installing other software.

    Go to and select the option appropriate for your system. Choose the Python 2.7 option (and not Python 3.x).

  • Install NCL, PyNIO, PyNGL, and other tools using "conda".

    This requires that step #2 be complete.

    First open a terminal window where you can type UNIX commands, and type:

    conda create --name ncl_test --channel dbrown --channel khallock ncl pyngl pynio matplotlib basemap ipython jupyter

    This creates a conda environment called "ncl_test" in which you can use during the tutorial and subsequently installs a bunch of software.

    To activate the "ncl_test" environment, type (from bash/sh):

    source activate ncl_test

    You can test NCL and matplotlib quickly by typing:

    python -c "from mpl_toolkits.basemap import Basemap"
    ncl -V ; ng4ex xy01n -clean

    The first command will echo something like "r16461", and the second command should pop up a window with an XY plot. Click on it with your left mouse button to make it go away.

    To test that you can run a jupyter notebook, type:

    jupyter notebook

  • Optional: install wgrib and wgrib2.

  • Install panoply

  • Optional: install ncview

  • Optional: install netcdf4python

    conda install netcdf4