Re: Increasing stacksize on Mac OSX

Memory allocation problem solved! The ultimate cause of the NCL malloc problem was inside a Fortran function that was being called by my NCL script, when I compiled the Fortran code using WRAPIT with the gfortran compiler on either Mac or Linux. This problem did not appear when I compiled the Fortran code using WRAPIT with the pgf90 compiler on Linux or the xlf90 compiler on AIX.

The Fortran function existed to find the indices of unique values of an array that was passed to it. An array 'inds' was allocated to be the size nX of the incoming array, and then filled with the nY indices of the unique values of the incoming array (therefore it had nX-nY values that weren't filled). A pointer array 'unique_inds' was also declared, but the gfortran malloc error arose from this assignment statement for 'unique_inds':

unique_inds = inds

In testing with gfortran I found that when all the values of the incoming array were unique, the malloc problem did not manifest itself, but when there were non-unique values in the array, the malloc error would eventually arise (but not usually on the first time the function was called during program execution). However, explicitly limiting the assignment to the filled values of 'inds' worked with gfortran (and still works with other compilers):

unique_inds = inds(1:nY)

It would appear that pgf90, ifort, xlf90, etc., all use the length of the pointer array as the amount of memory allocated, while gfortran uses the length of the data assigned as the amount of memory allocated. This was a tough bug for me to track down...

Jared

On Jun 30, 2011, at 11:21 AM, Jared Lee wrote:

> Wei,
>
> Here are my stats on bash:
>
> jaredlee$ ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> file size (blocks, -f) unlimited
> max locked memory (kbytes, -l) unlimited
> max memory size (kbytes, -m) unlimited
> open files (-n) 256
> pipe size (512 bytes, -p) 1
> stack size (kbytes, -s) 65532
> cpu time (seconds, -t) unlimited
> max user processes (-u) 266
> virtual memory (kbytes, -v) unlimited
>
> At any rate, given the nature of the malloc error, and after talking to a couple other people last night, it seems that the error is most likely a heap problem, rather than a stack problem, and that it's almost certainly a pointer problem. It seems like it's probably a bug within NCL, too, since it's triggering the malloc error. My NCL code calls NCL functions from other files, as well as functions from a couple Fortran 90 files. Do you have any suggestions for how to go about debugging this? This type of debugging is beyond my area of familiarity. I'm going to continue searching for help or suggestions on Google as well in the meantime...
>
> Jared
>
>
> On Jun 29, 2011, at 10:16 PM, Wei Huang wrote:
>
>> Jared,
>>
>> If you use CSH, you may try and see you memory limit as:
>>
>>> limit memoryuse 1000000
>>> limit
>> cputime unlimited
>> filesize unlimited
>> datasize unlimited
>> stacksize 8192 kbytes
>> coredumpsize 0 kbytes
>> memoryuse 1000000 kbytes
>> descriptors 1000000
>> memorylocked unlimited
>> maxproc 266
>>
>>
>>> limit memoryuse unlimit
>>> limit
>> cputime unlimited
>> filesize unlimited
>> datasize unlimited
>> stacksize 8192 kbytes
>> coredumpsize 0 kbytes
>> memoryuse unlimited
>> descriptors 1000000
>> memorylocked unlimited
>> maxproc 266
>>
>> Wei Huang
>> huangwei@ucar.edu
>> VETS/CISL
>> National Center for Atmospheric Research
>> P.O. Box 3000 (1850 Table Mesa Dr.)
>> Boulder, CO 80307-3000 USA
>> (303) 497-8924
>>
>>
>>
>>
>>
>> On Jun 29, 2011, at 6:57 PM, Jared Lee wrote:
>>
>>> I have 6 GB of memory on my Mac. I do believe that my script requires more memory than 64 MB. When I ran it on Linux systems I had unlimited stacksize and it ran without any problems.
>>>
>>> I came across this NCL for Mac page, which suggested making the stacksize unlimited. That should mean it's possible (unless whoever wrote this page never actually tried to do that themselves):
>>>
>>> http://www.ncl.ucar.edu/Download/macosx.shtml
>>>
>>> If I can't figure out how to increase the stacksize, that will likely cause me problems when I try to run WRF as well, because 64 MB is pretty darn small...
>>>
>>> Jared
>>>
>>>
>>> On Jun 29, 2011, at 6:47 PM, Dennis Shea wrote:
>>>
>>>> %> ulimit -s
>>>>
>>>> I think that the max stacksize for the MAC is 65532.
>>>> I don't think that there is an unlimited stacksize
>>>> like on many linux systems.
>>>>
>>>> ==
>>>> WHat is the available memory on your MAC? Does
>>>> your script require more memory then is available?
>>>>
>>>> On 6/29/11 6:35 PM, Jared Lee wrote:
>>>>> Hi, does anyone out there know how to increase the stacksize in Mac OSX (I'm running 10.6.8) beyond 65532 kB? I haven't been able to find a way to increase the stacksize to unlimited, or even anything larger than that. An NCL script that I have runs just fine on Linux systems, but when I run it on my new Mac, I get errors like this one:
>>>>>
>>>>> ==========
>>>>> ncl(47500) malloc: *** error for object 0x12c594c08: incorrect checksum for freed object - object was probably modified after being freed.
>>>>> *** set a breakpoint in malloc_error_break to debug
>>>>> sh: line 1: 47500 Abort trap
>>>>> ==========
>>>>>
>>>>> Because it's an mallloc error, I'm assuming that it'd go away if I could get the stacksize big enough, but OSX seems to make it hard or impossible to do that.
>>>>>
>>>>> Jared
>>>>>
>>>>> _______________________________________________
>>>>> ncl-talk mailing list
>>>>> List instructions, subscriber options, unsubscribe:
>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>> _______________________________________________
>>> ncl-talk mailing list
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Tue Jul 5 15:24:54 2011