Re: memory error question

From: <Jacob.Klee_at_nyahnyahspammersnyahnyah>
Date: Mon, 9 Jul 2007 15:38:23 -0400

Just a quick note:

We experienced some severe server outages at work today, including email.
If anyone replied to my post from earlier this morning and received a
bounced email, I would be very appreciative if the email could be resent.

My apologies.

-- Jacob Klee

                                                                           
             Jacob.Klee_at_dom.co
             m
             Sent by: To
             ncl-talk-bounces@ David Ian Brown <dbrown_at_ucar.edu>
             ucar.edu cc
                                       ncl-talk_at_ucar.edu
                                                                   Subject
             07/09/2007 09:59 Re: [ncl-talk] memory error
             AM question
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           

Hi David, and thank you for your reply.

So, my troubles are most likely related to the large print statement within
my innermost loop?

Over the weekend I let a version of the script run in which every variable
not already deleted was deleted before the end of the innermost do loop.
This had no impact on where the memory error occurred: in the midst of
printing the results of the 19th iteration of the innermost calculation
loop (I usually see the failure between ~45 and 80 iterations of the print
loop during the 19th iteration of the innermost calc. loop, with somewhere
61-65 being the most common).

As a restriction imposed by my analylsis software, I need my results
formatted as I have the print statement arranged. Additionally, short of
ending the script, I gather there is currently no way to release any of the
memory consumed by the multitude of unique strings I am producing. If this
is correct, what options do I have in making the calculations I need, and
still in the end (one way or another) ending with my data arranged
comparably to how I am printing it currently? (e.g. would writing the
results to a single NCL file be more efficient? If so, how would I then go
about writing the data out to a CSV file without again encountering the
same problems again?)

Thank you all again for your time.

-- Jacob Klee

             David Ian Brown
             <dbrown_at_ucar.edu>
             Sent by: To
             ncl-talk-bounces@ Dennis Shea <shea_at_ucar.edu>
             ucar.edu cc
                                       ncl-talk_at_ucar.edu,
                                       Jacob.Klee_at_dom.com
             07/06/2007 04:45 Subject
             PM Re: [ncl-talk] memory error
                                       question

>> In NCL, is there any reason if I delete every varaible (however that
>> deletion is acomplished) that 1,000,000 itterations of a script would
>> consume more memory than 1?

Hi Jacob,
I assume by "1,000,000 iterations" you are talking about some kind of
do loop inside a script -- not running a script that many times.

There is no significant use of memory in the iteration of a do loop
itself.
What can cause a problem, as has been discussed recently, is the
non-obvious
fact that every unique string created during the execution of an NCL
script is stored
internally for the life of the script. So for instance, the execution
of the loop below
creates 1,000,000 unique strings, leading to the consumption of around
18 Megabytes
of memory. This is not that big an amount of memory in the overall
scheme of things;
the bigger problem is that the hash table lookup takes more and more
time the more
strings it has to search through. In the case of the loop below it is
very noticeable that
it gets slower and slower as it progresses.

begin
do i = 1,1000
   do j = 1,1000
       print(i + " " + j)
   end do
end do
end

Changing the loop, as below, which reduces the number of unique strings
to 2000,
eliminates the slowdown, and reduces the memory consumption to a
scarcely noticeable
level.

begin
do i = 1,1000
   do j = 1,1000
       print("i = " + i )
       print("j = " + j )
   end do
end do
print ("end")
end

We hope to take a look at this to see if we can eliminate the permanent
storage of
transient strings.
  -dave

On Jul 5, 2007, at 9:21 AM, Dennis Shea wrote:

> [1] There is no way to delete multiple variables with
> one command. This capability has been requested before.
> It is on NCL's priority list. Unfortunately, NCL's request list
> is long :-(
>
> [2] If you have deleted all the temporary variables in a loop [let's
> say at the
> end of each iteration], then I don't think any extra memory
> should be used.
>
> There is the possibility of a memory leak but, to my knowledge,
> NCL has been pretty free of this problem. [One exception was
> about 3-4 years ago.]
>
> Jacob.Klee_at_dom.com wrote:
>> I will see what adjustments I can make.
>>
>> A thought came to me and I have not yet been able to fully research
>> it yet:
>> is there a way to delete / destroy all variables by a single command?
>>
>> In NCL, is there any reason if I delete every varaible (however that
>> deletion is acomplished) that 1,000,000 itterations of a script would
>> consume more memory than 1?
>>
>> -- Jacob
>>
>>
>>
>> Thanks for the information.
>>
>> It might help if you can deal with chunks of data at a time, and then
>> delete it. For example, if you need to write a bunch of variables to
>> a file, then rather than reading them all in at once and waiting until
>> the end to write them out, you can write them out one at a time, and
>> then delete them when you're done.
>>
>> It might even help if instead of doing everything in one big loop,
>> that you instead duplicate the loop and do a certain set of things
>> in one. This way you are not storing a whole bunch of variables
>> during the duration of a single loop iteration.
>>
>> --Mary
>>
>> On Tue, 3 Jul 2007 Jacob.Klee_at_dom.com wrote:
>>
>>
>>> Gladly.
>>>
>>> I am 'brute-force' data mining... I am creating a csv file of 5
>>> years
>>> worth of data from the North American Regional Reanalysis dataset
>>> (01Mar
>>> through 30Sep each year) and extracting / calculating numerous terms
>>>
>> which
>>
>>> may have some contribution / explanitory value concerning
>>> disruptions to
>>> our electric distribution network. (I curently have lightning data
>>> standing in as a placeholder for our outtage data which has yet to be
>>> compiled on the NARR grid.) I am going to feed the resulting file
>>> into
>>> DTREG and start the fun analyzing the data, and hopefully in the end
>>> producing a product which will further esitmate the potential impact
>>> of
>>> upcoming weather to our system.
>>>
>>>
>>> Also, the soil data is not involved in any calculations, but is
>>> writen
>>>
>> out
>>
>>> at the end of the file.
>>>
>>>
>>> -- Jacob
>>>
>>>
>>>
>>>
>>>
>>> Mary Haley
>>> <haley_at_ucar.edu>
>>> Sent by:
>>> To
>>> ncl-talk-bounces@ Jacob.Klee_at_dom.com
>>> ucar.edu
>>> cc
>>> 'ncl talk' <ncl-talk_at_ucar.edu>
>>>
>>> Subject
>>> 07/03/2007 12:35 Re: [ncl-talk] memory error
>>> PM question
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Hi Jacob,
>>>
>>> One of the places you can look if you start seeing memory allocation
>>> problems is if you are creating a lot of arrays, but not freeing them
>>> up with "delete" after you are done with them. You already have quite
>>> a few "delete" calls in your script, but it might be helpful if you
>>> could add more.
>>>
>>> I believe, yes, in this case you are running out of physical memory,
>>> and deleting some arrays should help.
>>>
>>> The killer might be the triple do loop in which you are calling
>>> addfile
>>> inside the innermost loop, and then reading a bunch of variables off
>>> the file.
>>>
>>> For example, you have code like this:
>>>
>>> SOILW00_01 = SOILW(0,:,:)
>>> SOILW01_04 = SOILW(1,:,:)
>>> SOILW04_10 = SOILW(2,:,:)
>>> SOILW10_20 = SOILW(3,:,:)
>>>
>>> in which I think SOILW is not being used after this point, so you
>>> could delete it.
>>>
>>> It might help us to know the intention of this program, so that
>>> we could suggest some other ways to do this kind of heavy file i/o
>>> and processing.
>>>
>>> Cheers,
>>>
>>> --Mary
>>>
>>>
>>> On Tue, 3 Jul 2007 Jacob.Klee_at_dom.com wrote:
>>>
>>>
>>>> Hello all,
>>>>
>>>> I have recived the error: fatal:NhlMalloc Failed:[errno=12]:Cannot
>>>>
>>> allocate
>>>
>>>> memory while running a memory intensive script. If my own memory
>>>> has
>>>>
>> not
>>
>>>> yet failed, I remember this translates to running out of physical
>>>>
>> memory?
>>
>>>> I am running the prerelease of ....3.0 for CYGWIN on an XP machine
>>>> with
>>>>
>>> 4gb
>>>
>>>> ram. I have included below the related files to my error.
>>>>
>>>> My questions are these: am I truly running out of memory? If so,
>>>> any
>>>> sugestions? If not, what is happening? (If anyone wants to run
>>>> this, I
>>>> would be happy to provide a handfull of the subseted NARR grib
>>>> files I
>>>>
>> am
>>
>>>> using.)
>>>>
>>>>
>>>>
>>>> -- Jacob Klee
>>>>
>>>> (See attached file: ncl.exe.stackdump)(See attached file:
>>>>
>>> DCP_trimmed.log)
>>>
>>>> (See attached file: DomConvectParams_test.ncl)(See attached file:
>>>> .hluresfile)
>>>>
>>>>
>>>> -----------------------------------------
>>>> CONFIDENTIALITY NOTICE: This electronic message contains
>>>> information which may be legally confidential and/or privileged and
>>>> does not in any case represent a firm ENERGY COMMODITY bid or offer
>>>> relating thereto which binds the sender without an additional
>>>> express written confirmation to that effect. The information is
>>>> intended solely for the individual or entity named above and access
>>>> by anyone else is unauthorized. If you are not the intended
>>>> recipient, any disclosure, copying, distribution, or use of the
>>>> contents of this information is prohibited and may be unlawful. If
>>>> you have received this electronic transmission in error, please
>>>> reply immediately to the sender that you have received the message
>>>> in error, and delete it. Thank you.
>>>>
>>>>
>>> _______________________________________________
>>> ncl-talk mailing list
>>> ncl-talk_at_ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>>
>>>
>>>
>>
>>
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk_at_ucar.edu
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>
>
> --
> ======================================================
> Dennis J. Shea tel: 303-497-1361 |
> P.O. Box 3000 fax: 303-497-1333 |
> Climate Analysis Section |
> Climate & Global Dynamics Div. |
> National Center for Atmospheric Research |
> Boulder, CO 80307 |
> USA email: shea 'at' ucar.edu |
> ======================================================
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk_at_ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
ncl-talk_at_ucar.edu
http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
ncl-talk_at_ucar.edu
http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
ncl-talk_at_ucar.edu
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Mon Jul 09 2007 - 13:38:23 MDT

This archive was generated by hypermail 2.2.0 : Wed Jul 11 2007 - 08:55:54 MDT