Re: working with memory

From: Mary Haley <haley_at_nyahnyahspammersnyahnyah>
Date: Wed, 18 Jan 2006 14:17:54 -0700 (MST)

On Mon, 16 Jan 2006, Micah Sklut wrote:

> Hi,
>
> I am creating plots of different regions along the US coastline.
> I am reading in 29 data files (which vary in time), joining them
> together,and then plotting
> the different regions per time. So, for example, there are 29 plots for the
> delmarva coastline, 29 plots for
> the new jersey coastline, etc.
>
> When I try to plot all the regions in the same program I am encountering
> memory problems. I get the following error:
> Fatal: NhlMalloc Failed: [errno=12]: Cannot allocate Memory
>
> If I only plot one region, then I am successful in creating all 29 plots. I
> am wondering if there is anything I can do to allocate more memory, so that
> I don't have to run individual programs for each region?
>
> Thanks.
>
> I am currently working with a Linux box using 384 MB ram.
> Program is below: (I load the same plot routines for each region
> - nam_graphs.ncl)

Hi Micah,

I don't know if anybody responded to you separately, so I'll take a
stab at it.

Generally, if you get that malloc memory error for NCL, it means that
you've hit up against the memory limit on your system. This happens
in NCL when you create lots of variables and never free them, or if
you are creating some really big variables and you don't have the
space for them.

In your case, it looks like you may have a lot of variables that are
being created. My suggestion would be to try to free up the variables
after you no longer need them, by using the "delete(x)" call. I don't
mean doing this for scalar variables, like "lonmin", but for array
variables that you may be creating inside the loop and not freeing up.

Also, since "do" loops can cause NCL to run slower (as with any
scripting language), you want to try to keep stuff out of a do loop as
much as possible (this is another potential memory problem).

In your case, some small things you can do include taking the code:

do i=0,28
hr = i*3

if (hr .lt. 10) then
   ext = "0"+hr
else
   ext = hr
end if
...
end do

and moving it outside the do loop as follows:

   nfiles = 29 ; or dimsizes(nam_files)
   hr = 3*ispan(0,nfiles-1,1)
   ext = sprinti("%02d",hr)
   do i=0,nfiles-1
      ...
   end do

Also, I noticed that you are loading scripts inside the do loop. I'm
not sure if this is an issue, but you might try moving these load
statements outside the loop at putting them at the beginning of the
script. I'm actually surprised it didn't complain about having these
load statements inside the loop.

These little suggestions may help with the memory problem. If not,
then if possible, could you send me the full script and I'll see
if there are any other memory issues.

--Mary

> begin
>
> nam_files = systemfunc("ls ./data/*.tm00")
> f = addfiles(nam_files+".grb","r")
> ListSetType(f,"join")
>
> u = f[:]->U_GRD_218_HTGL * 1.944
> v = f[:]->V_GRD_218_HTGL * 1.944
> lt = f[0]->gridlat_218
> ln = f[0]->gridlon_218
>
> ; convert u and v into speed
> spd = (u^2+v^2)^(.5)
>
> u_at_lat2d = lt
> u_at_lon2d = ln
> v_at_lat2d = lt
> v_at_lon2d = ln
> spd_at_lat2d = lt
> spd_at_lon2d = ln
>
> ;plot
>
> do i = 0,28
>
> hr = i*3
>
> if (hr .lt. 10) then
> ext = "0"+hr
> else
> ext = hr
> end if
>
> name_out="wna_stx/wna_stx_wind"+ext
> latmin = 25.512
> lonmin = -98.5
> latmax = 28.4
> lonmax = -94.5
> load "$SWELLINFO_ROOT/wind/nam_graphs.ncl"
>
> name_out="wna_ntx/wna_ntx_wind"+ext
> latmin = 27.3
> lonmin = -97.
> latmax = 30.188
> lonmax = -93.
> load "$SWELLINFO_ROOT/wind/nam_graphs.ncl"
>
> name_out="wna_flpan/wna_flpan_wind"+ext
> latmin = 28.
> lonmin = -88.
> latmax = 30.888
> lonmax = -84.
> load "$SWELLINFO_ROOT/wind/nam_graphs.ncl"
>
> .....(more regions).....
>
> end do
>
> end
>
_______________________________________________
ncl-talk mailing list
ncl-talk_at_ucar.edu
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Wed Jan 18 2006 - 14:17:54 MST

This archive was generated by hypermail 2.2.0 : Wed Jan 18 2006 - 14:34:50 MST