Re: Grib to netCDF conversion

From: Mary Haley (haley AT XXXXXX)
Date: Mon Apr 14 2003 - 15:28:57 MDT

  • Next message: Mary Haley: "MacOSX binaries available for version 4.2.0.a028"

    > Hi,
    > Initially I was extracting 4 variables from Eta 12 km grib files, now I'm
    > trying to extract 11 additional variables. I
    > editted the script, and it worked, but the run time went from ~1 minute to
    > ~30 minutes per job. Here's the code -
    > ; Set the variables to be converted to netcdf
    > pnames=(/"PRES","TMP","U_GRD","V_GRD","ULWRF","USWRF","DLWRF","DSWRF","SHTFL","LHTFL","WTMP","T_CDC","A_PCP","R_H","PRMSL"/)
    > plevels=(/"SFC","HTGL","HTGL","HTGL","SFC","SFC","SFC","SFC","SFC","SFC","SFC","EATM","SFC_acc3h","HTGL","MSL"/)
    > ;pvalues=(/1,11,33,34,/)
    > ncnames=(/"press","temp","uwind","vwind","ulwrf","uswrf","dlwrf","dswrf","shtfl","lhtfl","wtmp","tcdc","apcp","rh","prmsl"/)
    > ; Special variables of lat, lon no plevel segment
    > vlat= "gridlat_"+gnum
    > netcdf_out->lat = grib_in->\$vlat\$;
    > vlon= "gridlon_"+gnum
    > netcdf_out->lon = grib_in->\$vlon\$;
    > ; loop looking for matching variable names
    > do j=0,dimsizes(ncnames)-1
    > vnamej=pnames(j)+"_"+gnum+"_"+plevels(j)
    > do i=0,nNames-1
    > ; test for vnamej=grib_in->\$vNames(i)\$
    > if (vnamej.eq.vNames(i)) then
    > print ("Match "+j+" "+i+" "+vnamej+" "+vNames(i)+"
    > "+ncnames(j))
    > netcdf_out->\$ncnames(j)\$ = grib_in->\$vNames(i)\$
    > delete(grib_in->\$vNames(i)\$)
    > delete(netcdf_out->\$ncnames(j)\$)
    > end if
    > end do
    > end do
    > I've noticed that the script progressively takes longer to extract the
    > variables as it goes down the list. One of my coworkers thought that this
    > may be due to each variable taking additional memory. I tried to use
    > "delete" and "undef" to clear the variables from memory, but the script
    > still took a long time. Any ideas? Thanks.
    > Zack

    Hi Zack,

    I strongly recommend taking Sylvia's suggestion of reviewing how to
    write netCDF files "efficiently" from the URL she pointed out:

    It may look tedious at first, but pre-defining the variables in this
    matter, and especially the coordinate array information, can
    significantly improve the speed. If you have any questions about this,
    let us know in a private email.

    FYI, in your code snippet above, you have:


    You don't want to do this, because I believe this will delete the
    variable from the file you just wrote it to.

    Since you are not reading any variables off the file into a local
    file, there's no need to do any "delete" calls, so you can also


    Part of the slow down is also due to the nested do loops. The more do
    loops you have and the more code they contain, the more slow down you
    will see. We encourage users to avoid do loops where possible.

    If you are not interested in the "efficient" way of writing netCDF
    files, then you can trim your two do loops down into one do loop
    as follows (please note that this typed off-the-cuff, so there
    may be some syntax errors):

        vlat= "gridlat_"+gnum
        netcdf_out->lat = grib_in->\$vlat\$;
        vlon= "gridlon_"+gnum
        netcdf_out->lon = grib_in->\$vlon\$;
    ; Get list of potential variable names. Note that I'm doing this
    ; outside the do loop.
       vnamesF = pnames+"_"+gnum+"_"+plevels

     ; loop looking for matching variable names
       do i=0,nNames-1

    ; Get index in vnamesF where vNames(i) is equal to one of its values.
    ; If it's not missing, then we know we've found a match.
          index = ind(vNames(i).eq.vnamesF)

          if(.not.ismissing(index)) then
            netcdf_out->\$ncnames(index)\$ = grib_in->\$vNames(i)\$
          end if
       end do
    ncl-talk mailing list
    ncl-talk AT

    This archive was generated by hypermail 2b29 : Mon Apr 14 2003 - 15:57:43 MDT