Re: Problem reading a 4 Gb variable into memory.

Hi Doug & Dave,

Thanks very much for your replies.

Doug - I can confirm the ncl executable was built 64-bit on all the systems I tested.

Dave - Switching to 6.1.0-beta fixed my problem on mirage!

Huge props to the NCL development team for having already fixed this issue in the 6.1.0 beta :) This has been another ncl-talk success story.

Thanks for your help,

-Mike.

On Jun 20, 2012, at 6:39 PM, David Brown wrote:

> Hi Mike,
> NCL 6.0.0 has a bug that prevents variables larger than 2GB being read from GRIB files. This is fixed in 6.1.0-beta. Hopefully you can verify this on mirage by setting
> NCARG_ROOT to /contrib/ncl-6.1.0/ and then adding $NCARG_ROOT/bin to your PATH.
>
> Let us know if this works for you or not. Thanks.
> -dave
>
> On Jun 20, 2012, at 7:16 PM, Mike Pritchard wrote:
>
>> Hi,
>>
>> I have a ncl workflow that would benefit from reading in a very large variable from a grib file into memory but I am having difficulty doing so. The variable in question is a float of size (1280*1280*2*91*4) so 4.4 Gb in memory by my estimate, but ncl crashes with what I suspect is a memory violation on reading this variable in. I am 99% sure the issue is not due to lack of physical memory as the error has proved to be reproducable on several systems, all of which have more than enough memory (e.g. mirage @ CISL). I have also monitored the system memory usage using top while ncl is reading the variable, to verify that I am not bumping up against other users' or a hardware limit. At this stage, I'm pretty convinced the issue is software.
>> Any advice? A few more technical details follow....
>> Thanks,
>> -Mike.
>>
>> Here is what the standard out looks like, from a test on NICS Kraken's Cray (16Gb memory interactive environment), using ncl6.0.0:
>>
>> ncl 0> f=addfile ("2010012700_000_018_130128_135128_1279.yt.oper.an.ml.grb","r")
>> ncl 1> print (f)
>> ncl 2> T=f->T_GDS50_HYBL
>> fatal:NclMalloc Failed:[errno=12]
>> Segmentation fault (core dumped)
>>
>> Here is what gdb says if I parse the core file:
>>
>> (...)
>> Core was generated by `ncl'.
>> Program terminated with signal 11, Segmentation fault.
>> #0 0x00000000004cf3cf in GenericUnPack ()
>> (gdb)
>>
>>
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Wed Jun 20 20:19:00 2012