Re: Memory constraints?

Yeah, Gus, that's what they'd LIKE us to do. However, I can request a
single core/node and all 32GB on the node, and run it as a serial job.
Unless their LSF submission script, designed to protect users from
themselves, catches it and does something evil. I've caught them doing that
several times, already.

What I've done was a limit of 24GB and a single core on a single, reserved
node. That it still blew up was "curious".

I've got some ideas. Now.

Thanks, Gerry

On Wed, Jun 5, 2013 at 12:26 PM, Gus Correa <gus@ldeo.columbia.edu> wrote:

> Hi Gerry
>
> Check with your HPC wizards if they limit the memory per job per CPU.
> For instance, a solomonic HPC sys. admin. may think that
> the best practice in a 32GB cluster node with 8 CPU cores,
> is to allow a maximum of 4GB per CPU (maybe a bit less,
> if (s)he gives a bit of memory to the OS).
> In this scenario, if your job requests a single CPU,
> and your NCL script needs 2GB for each of three variables ...
> ... your script will break right on "dbz_a = new(..."
> If this is the case, you could request more CPUs or the whole node,
> just to get more memory.
>
> I hope this helps,
> Gus Correa
>
>
> On 06/05/2013 11:44 AM, Gerry Creager - NOAA Affiliate wrote:
> > 32 GB is available, Dennis. I was a little surprised, and wanted to make
> > sure there wasn't something here before I dig a little deeper. And, your
> > estimate confirms the calculations I'd also done offline.
> >
> > Adam, I'll try 'top'. I'll only violate a few constraints of the HPC
> > group's policies in the process...
> >
> > Thanks, Folks!
> > gerry
> >
> >
> > On Tue, Jun 4, 2013 at 6:16 PM, Dennis Shea <shea@ucar.edu
> > <mailto:shea@ucar.edu>> wrote:
> >
> >
> > Even a 32-bit version of NCL can handle 1.79GB variables.
> >
> > How much memory is available on the machine you are using?
> > ------------------------------__-----------------------------
> > dimx = 449l ; l means 64 bit signed integers
> > dimy = 399l
> > ens_size = 50l
> > dimz = 50l
> > vsize = dimx*dimy*dimz*ens_size ; variable size
> > bytes = 4*vsize ; bytes per variable
> > tsize = 3*bytes ; total size (3 variable)
> >
> > print("vsize="+vsize) ; 447,877,500
> > print("bytes="+bytes) ; 1,791,510,000
> > print("tsize="+tsize) ; 5,374,530,000
> >
> >
> > On 6/4/13 5:02 PM, Adam Phillips wrote:
> >
> > Hi Gerry,
> > I do not believe NCL has a memory limit. I've recently run a NCL
> > script
> > that took 75GB of memory at its' peak. This sounds like a machine
> > running out of memory, although your 3 arrays combined take just
> > short
> > of 6GB which isn't that much. You can run "top" in another
> window on
> > that machine to see what memory your job is taking (and what
> > memory is
> > available on that machine at that time).
> > Hope that helps.
> > Adam
> >
> > On 6/4/13 4:26 PM, Gerry Creager - NOAA Affiliate wrote:
> >
> > We're playing with a script that has the following snippet
> > of code:
> >
> > dimx = 449
> > dimy = 399
> > ens_size = 50
> > over_id = new((/10/),"graphic")
> >
> > x_start = 1
> > x_end = dimx-1
> > y_start = 1
> > y_end = dimy-1
> >
> > height = 1000.
> > dimz = 50
> >
> > Uwind_a = new((/dimz,dimy,dimx,ens_size/__),"float")
> > Vwind_a = new((/dimz,dimy,dimx,ens_size/__),"float")
> > dbz_a = new((/dimz,dimy,dimx,ens_size/__),"float")
> >
> > We're seeing an NclMalloc error after the last 'new'. Is
> > there a limit
> > in NCL I need to look at or do you think this is simple
> memory
> > exhaustion on the machine?
> >
> > Thanks, Gerry
> > --
> > Gerry Creager
> > NSSL/CIMMS
> > 405.325.6371 <tel:405.325.6371>
> > ++++++++++++++++++++++
> > "Big whorls have little whorls,
> > That feed on their velocity;
> > And little whorls have lesser whorls,
> > And so on to viscosity."
> > Lewis Fry Richardson (1881-1953)
> >
> >
> > _________________________________________________
> > ncl-talk mailing list
> > List instructions, subscriber options, unsubscribe:
> > http://mailman.ucar.edu/__mailman/listinfo/ncl-talk
> > <http://mailman.ucar.edu/mailman/listinfo/ncl-talk>
> >
> >
> >
> >
> >
> > _________________________________________________
> > ncl-talk mailing list
> > List instructions, subscriber options, unsubscribe:
> > http://mailman.ucar.edu/__mailman/listinfo/ncl-talk
> > <http://mailman.ucar.edu/mailman/listinfo/ncl-talk>
> >
> >
> >
> >
> > --
> > Gerry Creager
> > NSSL/CIMMS
> > 405.325.6371
> > ++++++++++++++++++++++
> > “Big whorls have little whorls,
> > That feed on their velocity;
> > And little whorls have lesser whorls,
> > And so on to viscosity.”
> > Lewis Fry Richardson (1881-1953)
> >
> >
> > _______________________________________________
> > ncl-talk mailing list
> > List instructions, subscriber options, unsubscribe:
> > http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>

-- 
Gerry Creager
NSSL/CIMMS
405.325.6371
++++++++++++++++++++++
“Big whorls have little whorls,
That feed on their velocity;
And little whorls have lesser whorls,
And so on to viscosity.”
Lewis Fry Richardson (1881-1953)

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Wed Jun 5 12:38:37 2013