Re: Memory leak with if statement and logical array

All,

Attached is a modified version of Susan's tst_memLeak script. I
deactivated all statements involving file I/O, leaving only simple
NCL statements and calls to NCL built-in functions. I also
commented out all if statements.

This version t3.ncl also shows a memory leak on Mac OS, though not
as severe as the one Susan describes. However, this one still has a
severe problem, running slower and slower with each iteration, which
is a killer for real processing applications.

My original post was about a memory leak with if statements, so this
is probably something different.

Susan, what kind of system are you running? Also, which NCL
version? It would be helpful to know whether these leaks occur on
platforms other than Mac OS.

--Dave A.
CU/CIRES Climate Diagnostics Center (CDC)
NOAA/ESRL/PSD, Climate Analysis Branch (CAB)

O'Neill, Susan - Portland, OR wrote:
>
> Hi Dave & everyone,
>
> I too have been struggling with a memory leak and was very excited to
> see your message. Unfortunately, it did not fix things for me.
> Attached is a very simple NCL script that loops through 3 yrs of flx
> files (tst_memLeak.ncl). I removed all program logic except for loop
> controls, system calls to un-tar & clean-up files, and opening the flx
> files. As the program runs it steadily consumes more and more memory,
> even after adding the explicit "True" to my logical if statements. It
> starts out using approx 2% of memory and by the time it's run 24 months
> it's using 27% of memory.
>
> When I have actual code in the script (see attached
> extractFlx_Montana.ncl), by the time 2 yrs have run 85% of memory is
> being used and it takes hours to process through one flx file.
>
> I can work around this by creating a wrapper script that only runs NCL a
> month at a time using ENV variables to set loop controls, but would
> really like to find a solution to this because I may want more data in
> memory to do graphing or other data manipulations in the future.
>
> Any insights/help/comments would be very appreciated! I am new to NCL
> so could quite possibly be missing something too, thank you!
>
> Susan

; 2007-apr-23 tst_memLeak.ncl Original problem demo, by Susan O'Neill.
; 2007-apr-23 t2.ncl Omit system() calls, addfile, load statements.
; Fix minor type mismatch in days_in_month call.
; 2007-apr-23 t3.ncl Omit all if statements.

;load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
;load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
;load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"

begin

      ; Define NARR files to process and time variables
      dir = "/home/smo/NARR"
      begYr = 1800
      endYr = 9999
      begMonth = 1
      endMonth = 12
      hrs_3 = (/"00", "03", "06", "09", "12", "15", "18", "21"/)

      ; calculate loop controls
      numYrs = endYr - begYr + 1
      numMonths = endMonth - begMonth + 1
      numHrs_day = dimsizes (hrs_3)

      do y = 0, numYrs - 1
         cyr = begYr + y
         
         print(""+cyr)

         do m = 1, numMonths
; if ( m .lt. 10 ) then
               cmon = "0" + m
; else
; cmon = m
; end if

;;; numDays = days_in_month ( stringtointeger (cyr), stringtointeger (cmon) )
            numDays = days_in_month ( cyr, stringtointeger (cmon) )

            ; Do system calls to un-tar the files for the month
; system ( "tar -xf " + dir + "/NARR_" + cyr + "/" + cyr + cmon + "/NARRflx_" + \
; cyr + cmon + "_01*.tar -C "+dir+"/NARR_"+cyr+"/"+cyr+cmon+"/" )
; system ( "tar -xf " + dir + "/NARR_" + cyr + "/" + cyr + cmon + "/NARRflx_" + \
; cyr + cmon + "_09*.tar -C "+dir+"/NARR_"+cyr+"/"+cyr+cmon+"/" )
; system ( "tar -xf " + dir + "/NARR_" + cyr + "/" + cyr + cmon + "/NARRflx_" + \
; cyr + cmon + "_17*.tar -C "+dir+"/NARR_"+cyr+"/"+cyr+cmon+"/" )
; system ( "tar -xf " + dir + "/NARR_" + cyr + "/" + cyr + cmon + "/NARRflx_" + \
; cyr + cmon + "_25*.tar -C "+dir+"/NARR_"+cyr+"/"+cyr+cmon+"/" )

            do d = 1, numDays
; if ( d .lt. 10 ) then
                  cday = "0"+d
; else
; cday = d
; end if

               do h = 0, numHrs_day - 1

                  chour = hrs_3(h)

                  ; ASSIGN FILENAMES TO VARIABLES
                  narrinfileFlx = dir+"/NARR_"+cyr+"/"+cyr+cmon+"/merged_AWIP32."+cyr+cmon+cday+chour+".RS.flx"
      
                  ; PRINT TO SCREEN TO MAKE SURE THE STRING IS CORRECT AND TO TRACK
; print("FileFlx="+narrinfileFlx)

; if (isfilepresent (narrinfileFlx) .eq. True) then
         
                     ; NEED TO APPEND THE '.grb' TO VARIABLE STRING SO NCL KNOWS FILE TYPE
                     narrinfileFlx = narrinfileFlx+".grb"
         
                     ; OPEN THE FILES
; narrinFlx = addfile (narrinfileFlx,"r")

; end if

               end do ; hour

            end do ; day

            ; remove files for the month
; system ( "rm "+dir+"/NARR_"+cyr+"/"+cyr+cmon+"/merged_AWIP32."+cyr+cmon+"*.RS.flx" )
         end do ; month

      end do ; year
end

_______________________________________________
ncl-talk mailing list
ncl-talk_at_ucar.edu
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Mon Apr 23 2007 - 19:13:33 MDT