Re: NCL operationally

From: Dennis Shea <shea_at_nyahnyahspammersnyahnyah>
Date: Mon, 19 Jan 2009 09:18:32 -0700

Hi Oliver

Today is a national holiday. Here in Colorado,
we are expecting near record breaking tempertures
[forecast high temperature is 16C :-) ]
Hence, I do not expect people to be logging in today.

I will let NCL's graphics people answer about future
png plans.

Comment: the reason why reading the grib1 or grib2
via NCL can take some time is that NCL is making
the variables on the GRIB files conform to a netCDF
data model. The issue is that GRIB is a *record format*
NOT a *file format* like netCDF or HDF. A GRIB file
is just a collection of individual grib records. There
are no rules. The grib records can be in any random
order. So NCL must look at the *entire file*. On the
fly, it creates pointers to records, creates uniqu
look like a netCDF file's contents. Importing to this
common data model comes at a price .... time to organize.

So, I doubt it is the overhead involved with using
"convert" that is the problem. It is the file
organization overhead that causes the slowdown.

I have never used MetView. I speculate it reads
'just' the grib records necessary for the plots.
A possible alternative to using ncl_convert2nc offline:
Use  wgrib to strip off the required records
and then have NCL work on the selected subset.
Something along the following:
    system('"wgrib -s grib_file | grep ":HGT:500 mb:" | wgrib -i -grib \
           grib_file -o temporary_grib_file" ')
    f = addfile ("temporary_grib_file.grb", "r")
    x = f->HGT...
      [do whatever]
    system("/bin/rm -f temporary_grib_file.grb"
As far as creating png via convert, my style is the following:
   pltDir  = "./"
   pltType = "png"
   pltName = "mjo_spectra"
   pltConvert = "-density 96 -trim +repage"
These could be done via NCL's Command Line Arguments if desired
   if (pltType.eq."png") then
       pltTypeLocal = "ps"
       pltTypeLocal = pltType
   end if
   pltPath = pltDir+pltName
   wks = gsn_open_wks(pltTypeLocal,pltPath)
    if (pltType.eq."png") then
        if (.not.isvar("pltConvert")) then
            pltConvert = " "               ; default
        end if
        system("convert "+pltConvert+" "+pltPath+".ps "+pltPath+".png")
        system("/bin/rm -f "+pltPath+".ps")
    end if
D wrote:
> Hi all,
> We are currently benchmarking NCL against MetView (from ECMWF). The goal 
> would be to use NCL operationally to produce plots of our model output. 
> Most of these products are produced in time-critical mode.
> Currently, there seem to be two bottlenecks that are slowing NCL down 
> considerably as compared to MetView. The first is the reading of our 
> model output in grib1 format. After offline conversion using 
> ncl_convert2nc to NetCDF format and execution of the same script simply 
> reading a couple of fields from a couple of files, the script runs 10x 
> faster. The second is the generation of raster graphics using NCL. A lot 
> of our plots are disseminated in PNG or GIF format. MetView can directly 
> write these formats whereas NCL writes a PS or PDF and the raster format 
> is produced using the "convert" utility.
> 1) I would be very much interested in future plans for NCL concerning 
> these two points.
> 2) Would a generation of NCGM and conversion using ctrans to a raster 
> format bring any gain in performance?
> Thanks and kind regards,
> Oliver
> ________________________________________
> Oliver Fuhrer
> Numerical Models
> Federal Departement of Home Affairs FDHA
> Federal Office of Meteorology and Climatology MeteoSwiss
> Kraehbuehlstrasse 58, P.O. Box 514, CH-8044 Zurich, Switzerland
> Tel. +41 44 256 93 59
> Fax  +41 44 256 92 78
> - First-hand information
> ------------------------------------------------------------------------
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
Received on Mon Jan 19 2009 - 09:18:32 MST

This archive was generated by hypermail 2.2.0 : Wed Jan 28 2009 - 09:24:27 MST