Re: memory allocation

From: Wei Huang <huangwei_at_nyahnyahspammersnyahnyah>
Date: Mon Nov 05 2012 - 15:38:58 MST

Noel,

In this case, I am wondering if you can select the year first, without reading the data
(or just read the data about years). Then you read the data for the selected year.
That way, you probably only need to read < 1Gb data.

Wei

huangwei@ucar.edu
VETS/CISL
National Center for Atmospheric Research
P.O. Box 3000 (1850 Table Mesa Dr.)
Boulder, CO 80307-3000 USA
(303) 497-8924

On Nov 5, 2012, at 2:44 PM, Noel Aloysius wrote:

> Hi Wei,
>
> Here is what I am trying to do...
>
> I want to read 3-hourly global land surface precipitation and temperature values for 59 years into to two arrays. The array size would be (/172400,180,360/) each. I, then, want to extract values for a randomly selected year - array size would be (/2928 or 2920, 180, 360/), perform some calculations in conjunction with monthly precip and temperature values and write the output as binary.
>
> I can get around by just reading only one year at time - that is what I am doing now.
>
> Since I don't have much restrictions on the amount of memory I can get on a cluster (upto 1000gb) I attempt to read all the 3-hourly values into the memory and continue with my calculations. I guess that is not possible.
>
> Would there be a way to hold an array of size mentioned above at different nodes on the cluster and perform calculations at the worker node?
>
> Thanks,
> Noel
>
>
> Noel Aloysius
>
>
> On Mon, Nov 5, 2012 at 4:20 PM, Wei Huang <huangwei@ucar.edu> wrote:
> Noel,
>
> You said that the 320gb memory is over 5 nodes.
> I guess you have a Linux Cluster, if that is the case,
> NCL can only use the memory from one node,
> which means, about 64gb.
>
> Lots of times, you may not need such a big array
> for your data, as you may only process one part
> of the array. In which case, you may only need
> a portion of the memory allocated first.
>
> Again, please give us more details, and we'll try to help.
>
> Wei
>
> huangwei@ucar.edu
> VETS/CISL
> National Center for Atmospheric Research
> P.O. Box 3000 (1850 Table Mesa Dr.)
> Boulder, CO 80307-3000 USA
> (303) 497-8924
>
>
>
>
>
>
> On Nov 5, 2012, at 12:24 PM, Noel Aloysius wrote:
>
>> Hi,
>>
>> I am using NCL 6.1.0. The system is Redhat Enterprise 64-bit and I have 320gb exclusively allocated over 5 nodes.
>>
>> I am able to create one array that takes 40gb. The Malloc error is thrown when I try to initialize the second array.
>>
>> Thanks,
>> Noel
>>
>>
>> Noel Aloysius
>>
>>
>> On Mon, Nov 5, 2012 at 12:35 PM, Wei Huang <huangwei@ucar.edu> wrote:
>> Noel,
>>
>> First, to use large memory you need NCL 6.0 or later, and make sure it is 64bit built.
>> Second, your machine has 320gb, does not mean mean each individual user can use that amount.
>> You need to check with your system admin that you can use up to that amount of memory.
>>
>> Any jobs manipulate that amount of data/memory will be slow, and very slow in I/O,
>> hope you have already thought of this.
>>
>> Please give us more details, and we will try to help you.
>>
>> Wei
>>
>> huangwei@ucar.edu
>> VETS/CISL
>> National Center for Atmospheric Research
>> P.O. Box 3000 (1850 Table Mesa Dr.)
>> Boulder, CO 80307-3000 USA
>> (303) 497-8924
>>
>>
>>
>>
>>
>>
>> On Nov 5, 2012, at 10:17 AM, Noel Aloysius wrote:
>>
>>> Hi NCL-talk,
>>>
>>> I want to load two datasets that will take a total of 90gb memory and do some processing. The total memory available is 320gb.
>>>
>>> Is it possible to do this. I tried but got the NclMalloc error.
>>>
>>> Thanks,
>>> Noel
>>>
>>>
>>>
>>> Noel Aloysius
>>> _______________________________________________
>>> ncl-talk mailing list
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Mon Nov 5 15:39:07 2012

This archive was generated by hypermail 2.1.8 : Tue Nov 06 2012 - 15:05:49 MST