Re: memory error question

[1] There is no way to delete multiple variables with
     one command. This capability has been requested before.
     It is on NCL's priority list. Unfortunately, NCL's request list is
long :-(

[2] If you have deleted all the temporary variables in a loop [let's say
at the
     end of each iteration], then I don't think any extra memory
     should be used.

     There is the possibility of a memory leak but, to my knowledge,
     NCL has been pretty free of this problem. [One exception was
     about 3-4 years ago.]

Jacob.Klee_at_dom.com wrote:
> I will see what adjustments I can make.
>
> A thought came to me and I have not yet been able to fully research it yet:
> is there a way to delete / destroy all variables by a single command?
>
> In NCL, is there any reason if I delete every varaible (however that
> deletion is acomplished) that 1,000,000 itterations of a script would
> consume more memory than 1?
>
> -- Jacob
>
>
>
> Thanks for the information.
>
> It might help if you can deal with chunks of data at a time, and then
> delete it. For example, if you need to write a bunch of variables to
> a file, then rather than reading them all in at once and waiting until
> the end to write them out, you can write them out one at a time, and
> then delete them when you're done.
>
> It might even help if instead of doing everything in one big loop,
> that you instead duplicate the loop and do a certain set of things
> in one. This way you are not storing a whole bunch of variables
> during the duration of a single loop iteration.
>
> --Mary
>
> On Tue, 3 Jul 2007 Jacob.Klee_at_dom.com wrote:
>
>
>> Gladly.
>>
>> I am 'brute-force' data mining... I am creating a csv file of 5 years
>> worth of data from the North American Regional Reanalysis dataset (01Mar
>> through 30Sep each year) and extracting / calculating numerous terms
>>
> which
>
>> may have some contribution / explanitory value concerning disruptions to
>> our electric distribution network. (I curently have lightning data
>> standing in as a placeholder for our outtage data which has yet to be
>> compiled on the NARR grid.) I am going to feed the resulting file into
>> DTREG and start the fun analyzing the data, and hopefully in the end
>> producing a product which will further esitmate the potential impact of
>> upcoming weather to our system.
>>
>>
>> Also, the soil data is not involved in any calculations, but is writen
>>
> out
>
>> at the end of the file.
>>
>>
>> -- Jacob
>>
>>
>>
>>
>>
>> Mary Haley
>> <haley_at_ucar.edu>
>> Sent by: To
>> ncl-talk-bounces@ Jacob.Klee_at_dom.com
>> ucar.edu cc
>> 'ncl talk' <ncl-talk_at_ucar.edu>
>> Subject
>> 07/03/2007 12:35 Re: [ncl-talk] memory error
>> PM question
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Hi Jacob,
>>
>> One of the places you can look if you start seeing memory allocation
>> problems is if you are creating a lot of arrays, but not freeing them
>> up with "delete" after you are done with them. You already have quite
>> a few "delete" calls in your script, but it might be helpful if you
>> could add more.
>>
>> I believe, yes, in this case you are running out of physical memory,
>> and deleting some arrays should help.
>>
>> The killer might be the triple do loop in which you are calling addfile
>> inside the innermost loop, and then reading a bunch of variables off
>> the file.
>>
>> For example, you have code like this:
>>
>> SOILW00_01 = SOILW(0,:,:)
>> SOILW01_04 = SOILW(1,:,:)
>> SOILW04_10 = SOILW(2,:,:)
>> SOILW10_20 = SOILW(3,:,:)
>>
>> in which I think SOILW is not being used after this point, so you
>> could delete it.
>>
>> It might help us to know the intention of this program, so that
>> we could suggest some other ways to do this kind of heavy file i/o
>> and processing.
>>
>> Cheers,
>>
>> --Mary
>>
>>
>> On Tue, 3 Jul 2007 Jacob.Klee_at_dom.com wrote:
>>
>>
>>> Hello all,
>>>
>>> I have recived the error: fatal:NhlMalloc Failed:[errno=12]:Cannot
>>>
>> allocate
>>
>>> memory while running a memory intensive script. If my own memory has
>>>
> not
>
>>> yet failed, I remember this translates to running out of physical
>>>
> memory?
>
>>> I am running the prerelease of ....3.0 for CYGWIN on an XP machine with
>>>
>> 4gb
>>
>>> ram. I have included below the related files to my error.
>>>
>>> My questions are these: am I truly running out of memory? If so, any
>>> sugestions? If not, what is happening? (If anyone wants to run this, I
>>> would be happy to provide a handfull of the subseted NARR grib files I
>>>
> am
>
>>> using.)
>>>
>>>
>>>
>>> -- Jacob Klee
>>>
>>> (See attached file: ncl.exe.stackdump)(See attached file:
>>>
>> DCP_trimmed.log)
>>
>>> (See attached file: DomConvectParams_test.ncl)(See attached file:
>>> .hluresfile)
>>>
>>>
>>> -----------------------------------------
>>> CONFIDENTIALITY NOTICE: This electronic message contains
>>> information which may be legally confidential and/or privileged and
>>> does not in any case represent a firm ENERGY COMMODITY bid or offer
>>> relating thereto which binds the sender without an additional
>>> express written confirmation to that effect. The information is
>>> intended solely for the individual or entity named above and access
>>> by anyone else is unauthorized. If you are not the intended
>>> recipient, any disclosure, copying, distribution, or use of the
>>> contents of this information is prohibited and may be unlawful. If
>>> you have received this electronic transmission in error, please
>>> reply immediately to the sender that you have received the message
>>> in error, and delete it. Thank you.
>>>
>>>
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk_at_ucar.edu
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>>
>>
>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk_at_ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>

-- 
======================================================
Dennis J. Shea                  tel: 303-497-1361    |
P.O. Box 3000                   fax: 303-497-1333    |
Climate Analysis Section                             |
Climate & Global Dynamics Div.                       |
National Center for Atmospheric Research             |
Boulder, CO  80307                                   |
USA                        email: shea 'at' ucar.edu |
======================================================
_______________________________________________
ncl-talk mailing list
ncl-talk_at_ucar.edu
http://mailman.ucar.edu/mailman/listinfo/ncl-talk