Re: memory error question

Thanks for the information.

It might help if you can deal with chunks of data at a time, and then
delete it. For example, if you need to write a bunch of variables to
a file, then rather than reading them all in at once and waiting until
the end to write them out, you can write them out one at a time, and
then delete them when you're done.

It might even help if instead of doing everything in one big loop,
that you instead duplicate the loop and do a certain set of things
in one. This way you are not storing a whole bunch of variables
during the duration of a single loop iteration.

--Mary

On Tue, 3 Jul 2007 Jacob.Klee_at_dom.com wrote:

> Gladly.
>
> I am 'brute-force' data mining... I am creating a csv file of 5 years
> worth of data from the North American Regional Reanalysis dataset (01Mar
> through 30Sep each year) and extracting / calculating numerous terms which
> may have some contribution / explanitory value concerning disruptions to
> our electric distribution network. (I curently have lightning data
> standing in as a placeholder for our outtage data which has yet to be
> compiled on the NARR grid.) I am going to feed the resulting file into
> DTREG and start the fun analyzing the data, and hopefully in the end
> producing a product which will further esitmate the potential impact of
> upcoming weather to our system.
>
>
> Also, the soil data is not involved in any calculations, but is writen out
> at the end of the file.
>
>
> -- Jacob
>
>
>
>
>
> Mary Haley
> <haley_at_ucar.edu>
> Sent by: To
> ncl-talk-bounces@ Jacob.Klee_at_dom.com
> ucar.edu cc
> 'ncl talk' <ncl-talk_at_ucar.edu>
> Subject
> 07/03/2007 12:35 Re: [ncl-talk] memory error
> PM question
>
>
>
>
>
>
>
>
>
>
> Hi Jacob,
>
> One of the places you can look if you start seeing memory allocation
> problems is if you are creating a lot of arrays, but not freeing them
> up with "delete" after you are done with them. You already have quite
> a few "delete" calls in your script, but it might be helpful if you
> could add more.
>
> I believe, yes, in this case you are running out of physical memory,
> and deleting some arrays should help.
>
> The killer might be the triple do loop in which you are calling addfile
> inside the innermost loop, and then reading a bunch of variables off
> the file.
>
> For example, you have code like this:
>
> SOILW00_01 = SOILW(0,:,:)
> SOILW01_04 = SOILW(1,:,:)
> SOILW04_10 = SOILW(2,:,:)
> SOILW10_20 = SOILW(3,:,:)
>
> in which I think SOILW is not being used after this point, so you
> could delete it.
>
> It might help us to know the intention of this program, so that
> we could suggest some other ways to do this kind of heavy file i/o
> and processing.
>
> Cheers,
>
> --Mary
>
>
> On Tue, 3 Jul 2007 Jacob.Klee_at_dom.com wrote:
>
>>
>> Hello all,
>>
>> I have recived the error: fatal:NhlMalloc Failed:[errno=12]:Cannot
> allocate
>> memory while running a memory intensive script. If my own memory has not
>> yet failed, I remember this translates to running out of physical memory?
>>
>> I am running the prerelease of ....3.0 for CYGWIN on an XP machine with
> 4gb
>> ram. I have included below the related files to my error.
>>
>> My questions are these: am I truly running out of memory? If so, any
>> sugestions? If not, what is happening? (If anyone wants to run this, I
>> would be happy to provide a handfull of the subseted NARR grib files I am
>> using.)
>>
>>
>>
>> -- Jacob Klee
>>
>> (See attached file: ncl.exe.stackdump)(See attached file:
> DCP_trimmed.log)
>> (See attached file: DomConvectParams_test.ncl)(See attached file:
>> .hluresfile)
>>
>>
>> -----------------------------------------
>> CONFIDENTIALITY NOTICE: This electronic message contains
>> information which may be legally confidential and/or privileged and
>> does not in any case represent a firm ENERGY COMMODITY bid or offer
>> relating thereto which binds the sender without an additional
>> express written confirmation to that effect. The information is
>> intended solely for the individual or entity named above and access
>> by anyone else is unauthorized. If you are not the intended
>> recipient, any disclosure, copying, distribution, or use of the
>> contents of this information is prohibited and may be unlawful. If
>> you have received this electronic transmission in error, please
>> reply immediately to the sender that you have received the message
>> in error, and delete it. Thank you.
>>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk_at_ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
>
_______________________________________________
ncl-talk mailing list
ncl-talk_at_ucar.edu
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Tue Jul 03 2007 - 14:56:27 MDT