Re: monitoring memory use in NCL

Dear Stephen,

We've tried a similar thing in order to find out how many parallel NCL
jobs we can fit onto a machine without running the risk of any of the
jobs running out of memory at some point.

Concerning the two NCL jobs, we've seen exactly the same thing. It seems
that when finishing a plot NCL (I think this was for ps in our case)
uses the spawn command (inside NCAR graphics if I remember correctly).
This effectively leads to an exact copy of the NCL process (using twice
the memory!) each time a ps graphics is finished.

Concerning the monitoring of NCL memory usage, we've decided for not
doing this from inside NCL but using a watchdog script running in
parallel while the NCL job is executing. This - of course - has the
disadvantage of not knowing at what point in the NCL script how much
memory is being used.

Cheers,
Oli

 
________________________________________

Oliver Fuhrer
Numerical Models

Federal Departement of Home Affairs FDHA
Federal Office of Meteorology and Climatology MeteoSwiss

Kraehbuehlstrasse 58, P.O. Box 514, CH-8044 Zurich, Switzerland

Tel. +41 44 256 93 59
Fax +41 44 256 92 78
oliver.fuhrer@meteoswiss.ch
www.meteoswiss.ch - First-hand information
  
 

> -----Original Message-----
> From: ncl-talk-bounces@ucar.edu
> [mailto:ncl-talk-bounces@ucar.edu] On Behalf Of Stephen Wood
> Sent: Freitag, 1. Oktober 2010 04:04
> To: ncl-talk@ucar.edu
> Subject: monitoring memory use in NCL
>
> Hi
>
> I'm wanting to carefully monitor memory usage on a set of NCL
> scripts.
>
> My first attempt was to do something like this
>
> print ("I'm at this point")
> system("ps u")
> the problem is that the output of system calls is output
> before all the NCL print commands so if you have several of
> these report lines you can't match each report from the "ps
> -u" with the point in the script it belongs to
>
> My next try was
>
> print ("I'm here")
> psrep=systemfunc("ps u")
> print (psrep)
>
> this works better - several lines are reported like this
>
> (0) USER PID %CPU %MEM VSZ RSS TTY STAT
> START TIME COMMAND
> (1) woodsw 5970 0.2 0.0 14248 3628 pts/1 S+
> 08:43 0:00 /usr/bin/python ./main.py namelist_tst
> (2) woodsw 5971 89.7 1.9 215732 161936 pts/1 S+
> 08:43 0:03 ncl ./main.ncl
> (3) woodsw 5982 0.0 1.9 215732 156688 pts/1 S+
> 08:43 0:00 ncl ./main.ncl
> (4) woodsw 5983 0.0 0.0 3564 900 pts/1 R+
> 08:43 0:00 ps u
> (5) woodsw 27690 0.0 0.0 9916 2488 pts/1 Ss
> Sep30 0:00 bash
> (6) woodsw 27944 0.0 0.0 9920 2436 pts/3 Ss+
> Sep30 0:00 bash
>
> this is a bit verbose, but does what I want,. however, I
> don't understand why there are *two* lines (and hence two
> processes) shown running my program main.ncl
>
> , or an explanation of these two processes?
>
> So, is there a better way to do what I want, monitoring
> memory usage. I know I can watch with some sort of monitor
> program, either graphical (system-monitor) or text (top, or
> ps) but getting the ncl to do it means I can tag it to points
> in the program.
>
>
> Thanks, Stephen wood, NZ
>
>
>
> Dr Stephen W. Wood
> National Institute of Water and Atmospheric Research Ltd
> Lauder, Private Bag 50061
> Omakau, Central Otago
> NEW ZEALAND
>
> Phone: work +64-3-4400426, home +64-3-4473789
> Fax: work +64-3-4473348, home +64-3-4473789
> Email: work s.wood AT niwa.co.nz , home stephen AT
> centralkennels.co.nz
>
> NIWA is the trading name of the National Institute of Water &
> Atmospheric Research Ltd.
>
_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Fri Oct 1 00:45:53 2010