Re: data size issue

Sanjiv,
This message is generated in the variable aggregating routine where the number of bytes in the requested aggregation is compared with the biggest number that can be understood by the allocation routine.
This depends on the size of the size_t type: 4 bytes or 8 bytes. If size_t is 4 bytes the biggest possible number is 2147483647; if 8 bytes it's a very large number > 9e18. In practice, this message should only ever be
encountered when running in 32 bit mode where size_t is 4 bytes. Note that you can run 32-bit programs on most 64-bit systems, so it is possible that you are running a 32-bit version of NCL. You should quickly be able to
tell whether you are running a 32-bit version of NCL: just execute this statement:

ncl 1> print(sizeof("long"))

If you get the answer 4, then you have a 32-bit version of NCL. If you get the answer 8, then something is probably wrong. You might want to try to estimate the size of the aggregated variable you are reading. But we may have to take a look.
 -dave

On Jan 22, 2014, at 3:14 PM, Mary Haley <haley@ucar.edu> wrote:

> Sanjiv,
>
> Are you running with at least version 6.0.0 of NCL? What does "ncl -V" report?
>
> Perhaps another possibility is how many files you are passing to "addfiles". I'm hoping Dave Brown will weigh in here.
>
> Dave, do you know what this error message signifies:
>
>>> fatal:Aggregating variable trefmxav from file list variable f1 as specified would exceed maximum NCL variable size
>
> --Mary
>
> On Jan 22, 2014, at 2:50 PM, Sanjiv Kumar <skumar.water@gmail.com> wrote:
>
>> Hi Mary:
>>
>> COLA system administrator has informed me that COLA machines are 64-bit machines, i.e. I am trying to run the program on 64-bit machines.
>>
>> -Sanjiv Kumar
>>
>>
>>
>>
>> On Tue, Jan 21, 2014 at 1:20 PM, Mary Haley <haley@ucar.edu> wrote:
>> Hi Sanjiv,
>>
>> Ditto here on the late reply. I was in a workshop last week.
>>
>> The error you're getting looks like it's from running on a 32-bit machine. Is that the case? If so, there are restrictions on 32-bit systems that limits you to 2 GB variables.
>>
>> Please see our FAQ at:
>>
>> http://www.ncl.ucar.edu/FAQ/#mem_issues_001
>>
>> --Mary
>>
>> On Jan 15, 2014, at 4:23 PM, Sanjiv Kumar <skumar.water@gmail.com> wrote:
>>
>>> Hi Mary:
>>>
>>> Sorry for the late reply. Below is the error message i get when i run my script on COLA machines:
>>>
>>> ************************************************************************
>>>
>>> fatal:Aggregating variable trefmxav from file list variable f1 as specified would exceed maximum NCL variable size
>>> fatal:["Execute.c":8128]:Execute: Error occurred at or near line 46 in file VAR_ANA_TREFMXAV_MAYICS.ncl
>>>
>>> fatal:Variable (var1SP) is undefined
>>> fatal:["Execute.c":8128]:Execute: Error occurred at or near line 48 in file VAR_ANA_TREFMXAV_MAYICS.ncl
>>>
>>> fatal:Aggregating variable trefmxav from file list variable f2 as specified would exceed maximum NCL variable size
>>> fatal:["Execute.c":8128]:Execute: Error occurred at or near line 61 in file VAR_ANA_TREFMXAV_MAYICS.ncl
>>> ******************************************************************************
>>>
>>>
>>> However, when i run the same script on 'geyser', it runs fine. Both machines have same NCL version 6.1.2
>>>
>>> Let me know if any further details is needed. This time i will reply early.
>>>
>>> Best Regards
>>> Sanjiv Kumar
>>>
>>>
>>> On Thu, Jan 9, 2014 at 10:28 AM, Mary Haley <haley@ucar.edu> wrote:
>>> Hi Sanjiv,
>>>
>>> What do you mean by "it breaks down"? Do you actually get an error message, or does the script take a long time to run?
>>>
>>> Also, when you say "large data volume", do you mean you have a few really large variables, or several variables that take up more than 2 GB?
>>>
>>> It would help if we could see a script that illustrates the problem, and if we had an idea of the size of your variables (you can get this by using printVarSummary on your variables).
>>>
>>> If you are not sure where it is "breaking down", then you can try commenting out the "begin" and "end" statements in your main code (if any), and then running the script with:
>>>
>>> ncl -x your_script.ncl
>>>
>>> The "-x" will cause every line to be echoed as it is executed.
>>>
>>> --Mary
>>>
>>> On Jan 7, 2014, at 1:22 PM, Sanjiv Kumar <skumar.water@gmail.com> wrote:
>>>
>>>> Hello:
>>>>
>>>> Happy New Year!
>>>>
>>>> I am running a ncl script, that handles large data volume (i guess more than 2GB). The script runs fine when i use NCAR machines (geyser). However, when i run the same script on our own machines (COLA machines) using the same NCL version (6.1.2); it breaks down, most likely due to large data volume size.
>>>>
>>>> We are not sure how to fix this problem. Any suggestion/help is highly appreciated.
>>>>
>>>> Best Regards
>>>> Sanjiv Kumar
>>>>
>>>>
>>>> _______________________________________________
>>>> ncl-talk mailing list
>>>> List instructions, subscriber options, unsubscribe:
>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>>
>>> _______________________________________________
>>> ncl-talk mailing list
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Wed Jan 22 16:40:38 2014