Re: ncl performance

On Tue, 16 Jun 2009 Oliver.Fuhrer_at_meteoswiss.ch wrote:

> Hi Dennis,
>
> This might be a detail, but I think NCL is doing some multiprocessing...
> We were benchmarking NCL for some time and investigating on how to use
> it optimally on our dual core processors. Doing that we noted that NCARG
> actually uses the "fork" command when finishing up graphics and thus
> sometimes has 2 or even more (depending on how often your script is
> finishing up a graphics file) parallel jobs running all using the same
> amount of memory.
>
> This feature was actually rather annoying to us as it sometimes led the
> machine into swapping.
>
> Cheers,
> Oli

Hi Oli,

We're curious what is causing your code to fork, because we don't
think we have any explicit fork calls, except for when you run NCL
interactively.

You mentioned the problem happens when you are finishing up a graphics
file. Are you by any chance calling "system" or "systemfunc" to do a
graphical conversion? This will cause your system to spawn another
process.

--Mary

>
> ________________________________________
>
> Oliver Fuhrer
> Numerical Models
>
> Federal Departement of Home Affairs FDHA
> Federal Office of Meteorology and Climatology MeteoSwiss
>
> Kraehbuehlstrasse 58, P.O. Box 514, CH-8044 Zurich, Switzerland
>
> Tel. +41 44 256 93 59
> Fax +41 44 256 92 78
> oliver.fuhrer_at_meteoswiss.ch
> www.meteoswiss.ch - First-hand information
>
>
>
>> -----Original Message-----
>> From: ncl-talk-bounces_at_ucar.edu
>> [mailto:ncl-talk-bounces_at_ucar.edu] On Behalf Of Dennis Shea
>> Sent: Montag, 15. Juni 2009 18:37
>> To: Micah Sklut
>> Cc: NCL_Talk
>> Subject: Re: ncl performance
>>
>> Multithreading or multiprocessing or parallel processing??
>>
>> AFAIK multithreading is more system level. I don't think this
>> is user controllable.
>>
>> Multiprocessing: It is not done from the NCL level. The
>> user can spawn multiple jobs to run on different CPUs from
>> (say) a C-shell script. Of course, if you have only one
>> data channel per cpu to a common memory, the different processes
>> will still have IO 'conflicts' when reading/writing to a common
>> (shared) memory.
>>
>> Parallel processing: Requires 'load balancing'' to work well.
>> Some algorithms [eg: eigenvector conputations] could be set
>> up to use this and (possibly) some graphics
>> calculations but, currently, NCL does not include this capability.
>>
>> D
>>
>>
>>
>>
>>
>> Micah Sklut wrote:
>>> Dennis,
>>>
>>> Can NCL utilize multi threading? or is it only going to use one
>>> processor?
>>>
>>>
>>> On Fri, Jun 12, 2009 at 6:47 PM, Dennis Shea <shea_at_ucar.edu
>>> <mailto:shea_at_ucar.edu>> wrote:
>>>
>>> IMHO ... memory is always good to have ...
>>>
>>> Often mhz are emphasized ... not that important in and of
>>> itself. Overlooked issues are data movement from memory
>> and cache.
>>>
>>> You can have lots of memory and a fast cpu but if the data
>>> movement is slow then the code will run slowly. I do not
>>> just mean reading from memory ... writing takes time also.
>>> If you have intertwined read/write tasks then performance
>>> could be even slower.
>>>
>>> D
>>>
>>> Micah Sklut wrote:
>>>
>>> Hi NCL,
>>>
>>> Its been a while, since my last questions, but I believe I
>>> have a straight forward one here.
>>>
>>> What is the best way to increase performance for
>> ncl scripts.
>>> I have batch jobs where I read in large grib files,
>> and output
>>> lots of images.
>>>
>>> Will performance improve by increasing RAM ?
>>> Will performance improve by increasing Processor Speed?
>>>
>>> My guess, is that the memory is going to be the biggest
>>> factor, and then processor speed second.
>>>
>>> Thanks for your help!
>>>
>>> Micah
>>>
>> --------------------------------------------------------------
>> ----------
>>>
>>> _______________________________________________
>>> ncl-talk mailing list
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>>
>>>
>>>
>>> --
>>> ======================================================
>>> Dennis J. Shea tel: 303-497-1361 |
>>> P.O. Box 3000 fax: 303-497-1333 |
>>> Climate Analysis Section |
>>> Climate & Global Dynamics Div. |
>>> National Center for Atmospheric Research |
>>> Boulder, CO 80307 |
>>> USA email: shea 'at' ucar.edu
>>> <http://ucar.edu> |
>>> ======================================================
>>>
>>>
>>
>>
>> --
>> ======================================================
>> Dennis J. Shea tel: 303-497-1361 |
>> P.O. Box 3000 fax: 303-497-1333 |
>> Climate Analysis Section |
>> Climate & Global Dynamics Div. |
>> National Center for Atmospheric Research |
>> Boulder, CO 80307 |
>> USA email: shea 'at' ucar.edu |
>> ======================================================
>>
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Tue Jun 16 2009 - 13:25:57 MDT