Re: Memory constraints?

Hi Gerry

Check with your HPC wizards if they limit the memory per job per CPU.
For instance, a solomonic HPC sys. admin. may think that
the best practice in a 32GB cluster node with 8 CPU cores,
is to allow a maximum of 4GB per CPU (maybe a bit less,
if (s)he gives a bit of memory to the OS).
In this scenario, if your job requests a single CPU,
and your NCL script needs 2GB for each of three variables ...
... your script will break right on "dbz_a = new(..."
If this is the case, you could request more CPUs or the whole node,
just to get more memory.

I hope this helps,
Gus Correa

On 06/05/2013 11:44 AM, Gerry Creager - NOAA Affiliate wrote:
> 32 GB is available, Dennis. I was a little surprised, and wanted to make
> sure there wasn't something here before I dig a little deeper. And, your
> estimate confirms the calculations I'd also done offline.
>
> Adam, I'll try 'top'. I'll only violate a few constraints of the HPC
> group's policies in the process...
>
> Thanks, Folks!
> gerry
>
>
> On Tue, Jun 4, 2013 at 6:16 PM, Dennis Shea <shea@ucar.edu
> <mailto:shea@ucar.edu>> wrote:
>
>
> Even a 32-bit version of NCL can handle 1.79GB variables.
>
> How much memory is available on the machine you are using?
> ------------------------------__-----------------------------
> dimx = 449l ; l means 64 bit signed integers
> dimy = 399l
> ens_size = 50l
> dimz = 50l
> vsize = dimx*dimy*dimz*ens_size ; variable size
> bytes = 4*vsize ; bytes per variable
> tsize = 3*bytes ; total size (3 variable)
>
> print("vsize="+vsize) ; 447,877,500
> print("bytes="+bytes) ; 1,791,510,000
> print("tsize="+tsize) ; 5,374,530,000
>
>
> On 6/4/13 5:02 PM, Adam Phillips wrote:
>
> Hi Gerry,
> I do not believe NCL has a memory limit. I've recently run a NCL
> script
> that took 75GB of memory at its' peak. This sounds like a machine
> running out of memory, although your 3 arrays combined take just
> short
> of 6GB which isn't that much. You can run "top" in another window on
> that machine to see what memory your job is taking (and what
> memory is
> available on that machine at that time).
> Hope that helps.
> Adam
>
> On 6/4/13 4:26 PM, Gerry Creager - NOAA Affiliate wrote:
>
> We're playing with a script that has the following snippet
> of code:
>
> dimx = 449
> dimy = 399
> ens_size = 50
> over_id = new((/10/),"graphic")
>
> x_start = 1
> x_end = dimx-1
> y_start = 1
> y_end = dimy-1
>
> height = 1000.
> dimz = 50
>
> Uwind_a = new((/dimz,dimy,dimx,ens_size/__),"float")
> Vwind_a = new((/dimz,dimy,dimx,ens_size/__),"float")
> dbz_a = new((/dimz,dimy,dimx,ens_size/__),"float")
>
> We're seeing an NclMalloc error after the last 'new'. Is
> there a limit
> in NCL I need to look at or do you think this is simple memory
> exhaustion on the machine?
>
> Thanks, Gerry
> --
> Gerry Creager
> NSSL/CIMMS
> 405.325.6371 <tel:405.325.6371>
> ++++++++++++++++++++++
> "Big whorls have little whorls,
> That feed on their velocity;
> And little whorls have lesser whorls,
> And so on to viscosity."
> Lewis Fry Richardson (1881-1953)
>
>
> _________________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/__mailman/listinfo/ncl-talk
> <http://mailman.ucar.edu/mailman/listinfo/ncl-talk>
>
>
>
>
>
> _________________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/__mailman/listinfo/ncl-talk
> <http://mailman.ucar.edu/mailman/listinfo/ncl-talk>
>
>
>
>
> --
> Gerry Creager
> NSSL/CIMMS
> 405.325.6371
> ++++++++++++++++++++++
> “Big whorls have little whorls,
> That feed on their velocity;
> And little whorls have lesser whorls,
> And so on to viscosity.”
> Lewis Fry Richardson (1881-1953)
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Wed Jun 5 11:26:49 2013