Re: monitoring memory use in NCL

Hi Stephen,
Hopefully without getting too technical, the reason you see 2 copies
of ncl executing is that systemfunc is implemented using the Unix
'fork' and 'pipe' system calls. fork creates a copy of the original
process. The original process is the parent and the new process is the
child. In the case of systemfunc the child makes a call to the
'system' call which creates another child process which happens to be
a Unix shell that executes whatever command you have passed in. When
the command completes, the first child process exits. However, through
the magic of 'pipe' all its output is channeled back to the original
parent ncl process which then returns this output as an array of
strings.

Given this information and a perusal of the 'ps' man page, I think it
is possible to get just the information you want from within NCL. Note
that, based on the description above, the shell command that runs the
ps utility is actually the grandchild of the original ncl process.
Also note that the 'ps' command can have different options and
semantics on different Unix systems, so the particular set of options
that works in my environment (Mac OS X) are probably not universal.

Anyway here is an incantation that gives only the command name (ncl),
the original ncl process id, parent process id, virtual size, resident
size, and percent memory used in a single string without any extra text:

ncl 37> print(systemfunc("ps -p `ps -p $PPID -o ppid=''` -o
command='',pid='',ppid='',vsz='',rss='',%mem=''"))
(0) ncl 4959 406 645572 4220 0.1

Note there is a 'ps' command embedded within a 'ps' command. Its
purpose is only to print the parent process id of the parent of the
executing process: $PPID is a bash shell internally defined
environment variable that returns the parent of the executing process.
-o ppid='' (2 single quotes) causes the ps command to output only the
parent process id field without any extra text.
So if you execute that part of the command by itself you just get the
process id of the original ncl process:

ncl 28> print(systemfunc("ps -p $PPID -o ppid=''"))
(0) 4959

This becomes the input to the -p option of the enclosing 'ps' command.

Also you can compare the first output with a command that is the same
except that it operates directly on the $PPID:

ncl 38> print(systemfunc("ps -p $PPID -o
command='',pid='',ppid='',vsz='',rss='',%mem=''"))
(0) ncl 5435 4959 645572 200 0.0
ncl 39> print(systemfunc("ps -p $PPID -o
command='',pid='',ppid='',vsz='',rss='',%mem=''"))
(0) ncl 5438 4959 645572 204 0.0

This shows the child ncl process only. You can see that each time you
execute it there is a new process id, but the parent process id stays
the same and it is the original process id. Also it is worth noting
that although the virtual size of the child is the same as the parent,
its resident size (actual space it occupies in your memory) is tiny
compared to the parent.

You may have to fiddle a bit and look at the 'ps' man page if you are
running on a different system.
Good luck.
  -dave

On Sep 30, 2010, at 8:03 PM, Stephen Wood wrote:

> Hi
>
> I'm wanting to carefully monitor memory usage on a set of NCL scripts.
>
> My first attempt was to do something like this
>
> print ("I'm at this point")
> system("ps u")
> the problem is that the output of system calls is output before all
> the NCL print commands so if you have several of these report lines
> you can't match each report from the "ps -u" with the point in the
> script it belongs to
>
> My next try was
>
> print ("I'm here")
> psrep=systemfunc("ps u")
> print (psrep)
>
> this works better - several lines are reported like this
>
> (0) USER PID %CPU %MEM VSZ RSS TTY STAT START
> TIME COMMAND
> (1) woodsw 5970 0.2 0.0 14248 3628 pts/1 S+ 08:43
> 0:00 /usr/bin/python ./main.py namelist_tst
> (2) woodsw 5971 89.7 1.9 215732 161936 pts/1 S+ 08:43
> 0:03 ncl ./main.ncl
> (3) woodsw 5982 0.0 1.9 215732 156688 pts/1 S+ 08:43
> 0:00 ncl ./main.ncl
> (4) woodsw 5983 0.0 0.0 3564 900 pts/1 R+ 08:43
> 0:00 ps u
> (5) woodsw 27690 0.0 0.0 9916 2488 pts/1 Ss Sep30
> 0:00 bash
> (6) woodsw 27944 0.0 0.0 9920 2436 pts/3 Ss+ Sep30
> 0:00 bash
>
> this is a bit verbose, but does what I want,. however, I don't
> understand why there are *two* lines (and hence two processes) shown
> running my program main.ncl
>
> , or an explanation of these two processes?
>
> So, is there a better way to do what I want, monitoring memory
> usage. I know I can watch with some sort of monitor program, either
> graphical (system-monitor) or text (top, or ps) but getting the ncl
> to do it means I can tag it to points in the program.
>
>
> Thanks, Stephen wood, NZ
>
>
> Dr Stephen W. Wood
> National Institute of Water and Atmospheric Research Ltd
> Lauder, Private Bag 50061
> Omakau, Central Otago
> NEW ZEALAND
>
> Phone: work +64-3-4400426, home +64-3-4473789
> Fax: work +64-3-4473348, home +64-3-4473789
> Email: work s.wood AT niwa.co.nz , home stephen AT
> centralkennels.co.nz
> NIWA is the trading name of the National Institute of Water &
> Atmospheric Research Ltd.
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Fri Oct 1 12:18:56 2010