Re: memory error question

From: David Ian Brown <dbrown_at_nyahnyahspammersnyahnyah>
Date: Mon, 9 Jul 2007 15:56:39 -0600

Looking at your script, my first thought would be to try using the
write_matrix procedure to eliminate the big print statement at the end.
You could create a float array of 183 x <however many values you have
in your print statement>,
populate it with the values that your current print statement
outputs, and then output it to a file. Since, unfortunately,
write_matrix does not
currently have an option for appending to an existing file, you could
write to
a temporary file then use a statement like:

system("cat temp_file >> final_csv_file")

to get all the output into a single file.

Since it looks like almost all your output is floating point numbers it
seems like this would
work if you can live with a couple of integer values being written out
as floating point numbers.

If that approach does not seem suitable, the other option is to create
a shared object from
Fortran or C code. This approach would allow you to specify every
detail of how the data is output,
but would, of course, be a bit more work.

Another thing you could do is try to eliminate the innermost loop (do
N_INDEX = 0,182)

By converting the input data variables to 1D arrays you could use
vector subscripting to assign each
variable in one step.

Convert the 2D indexes into 1D:

Dom1d = DomY * <size of X dimension> + DomX

where presumably size of X dimension would be the size of g3_y_1
(confusingly) in the GRIB file.

Then if you set
APCPN1d = ndtooned(APCPN)

The values you are looking for would be selected with

APCPN1d(Dom1d)

  -dave

On Jul 9, 2007, at 7:59 AM, Jacob.Klee_at_dom.com wrote:

> Hi David, and thank you for your reply.
>
> So, my troubles are most likely related to the large print statement
> within
> my innermost loop?
>
> Over the weekend I let a version of the script run in which every
> variable
> not already deleted was deleted before the end of the innermost do
> loop.
> This had no impact on where the memory error occurred: in the midst of
> printing the results of the 19th iteration of the innermost calculation
> loop (I usually see the failure between ~45 and 80 iterations of the
> print
> loop during the 19th iteration of the innermost calc. loop, with
> somewhere
> 61-65 being the most common).
>
> As a restriction imposed by my analylsis software, I need my results
> formatted as I have the print statement arranged. Additionally, short
> of
> ending the script, I gather there is currently no way to release any
> of the
> memory consumed by the multitude of unique strings I am producing. If
> this
> is correct, what options do I have in making the calculations I need,
> and
> still in the end (one way or another) ending with my data arranged
> comparably to how I am printing it currently? (e.g. would writing the
> results to a single NCL file be more efficient? If so, how would I
> then go
> about writing the data out to a CSV file without again encountering the
> same problems again?)
>
> Thank you all again for your time.
>
>
>
> -- Jacob Klee
>
>
>
>
>
>
>
> David Ian Brown
> <dbrown_at_ucar.edu>
> Sent by:
> To
> ncl-talk-bounces@ Dennis Shea <shea_at_ucar.edu>
> ucar.edu
> cc
> ncl-talk_at_ucar.edu,
> Jacob.Klee_at_dom.com
> 07/06/2007 04:45
> Subject
> PM Re: [ncl-talk] memory error
> question
>
>
>
>
>
>
>
>
>
>
>>> In NCL, is there any reason if I delete every varaible (however that
>>> deletion is acomplished) that 1,000,000 itterations of a script would
>>> consume more memory than 1?
>
>
> Hi Jacob,
> I assume by "1,000,000 iterations" you are talking about some kind of
> do loop inside a script -- not running a script that many times.
>
> There is no significant use of memory in the iteration of a do loop
> itself.
> What can cause a problem, as has been discussed recently, is the
> non-obvious
> fact that every unique string created during the execution of an NCL
> script is stored
> internally for the life of the script. So for instance, the execution
> of the loop below
> creates 1,000,000 unique strings, leading to the consumption of around
> 18 Megabytes
> of memory. This is not that big an amount of memory in the overall
> scheme of things;
> the bigger problem is that the hash table lookup takes more and more
> time the more
> strings it has to search through. In the case of the loop below it is
> very noticeable that
> it gets slower and slower as it progresses.
>
> begin
> do i = 1,1000
> do j = 1,1000
> print(i + " " + j)
> end do
> end do
> end
>
> Changing the loop, as below, which reduces the number of unique strings
> to 2000,
> eliminates the slowdown, and reduces the memory consumption to a
> scarcely noticeable
> level.
>
> begin
> do i = 1,1000
> do j = 1,1000
> print("i = " + i )
> print("j = " + j )
> end do
> end do
> print ("end")
> end
>
> We hope to take a look at this to see if we can eliminate the permanent
> storage of
> transient strings.
> -dave
>
>
> On Jul 5, 2007, at 9:21 AM, Dennis Shea wrote:
>
>> [1] There is no way to delete multiple variables with
>> one command. This capability has been requested before.
>> It is on NCL's priority list. Unfortunately, NCL's request list
>> is long :-(
>>
>> [2] If you have deleted all the temporary variables in a loop [let's
>> say at the
>> end of each iteration], then I don't think any extra memory
>> should be used.
>>
>> There is the possibility of a memory leak but, to my knowledge,
>> NCL has been pretty free of this problem. [One exception was
>> about 3-4 years ago.]
>>
>> Jacob.Klee_at_dom.com wrote:
>>> I will see what adjustments I can make.
>>>
>>> A thought came to me and I have not yet been able to fully research
>>> it yet:
>>> is there a way to delete / destroy all variables by a single command?
>>>
>>> In NCL, is there any reason if I delete every varaible (however that
>>> deletion is acomplished) that 1,000,000 itterations of a script would
>>> consume more memory than 1?
>>>
>>> -- Jacob
>>>
>>>
>>>
>>> Thanks for the information.
>>>
>>> It might help if you can deal with chunks of data at a time, and then
>>> delete it. For example, if you need to write a bunch of variables to
>>> a file, then rather than reading them all in at once and waiting
>>> until
>>> the end to write them out, you can write them out one at a time, and
>>> then delete them when you're done.
>>>
>>> It might even help if instead of doing everything in one big loop,
>>> that you instead duplicate the loop and do a certain set of things
>>> in one. This way you are not storing a whole bunch of variables
>>> during the duration of a single loop iteration.
>>>
>>> --Mary
>>>
>>> On Tue, 3 Jul 2007 Jacob.Klee_at_dom.com wrote:
>>>
>>>
>>>> Gladly.
>>>>
>>>> I am 'brute-force' data mining... I am creating a csv file of 5
>>>> years
>>>> worth of data from the North American Regional Reanalysis dataset
>>>> (01Mar
>>>> through 30Sep each year) and extracting / calculating numerous terms
>>>>
>>> which
>>>
>>>> may have some contribution / explanitory value concerning
>>>> disruptions to
>>>> our electric distribution network. (I curently have lightning data
>>>> standing in as a placeholder for our outtage data which has yet to
>>>> be
>>>> compiled on the NARR grid.) I am going to feed the resulting file
>>>> into
>>>> DTREG and start the fun analyzing the data, and hopefully in the end
>>>> producing a product which will further esitmate the potential impact
>>>> of
>>>> upcoming weather to our system.
>>>>
>>>>
>>>> Also, the soil data is not involved in any calculations, but is
>>>> writen
>>>>
>>> out
>>>
>>>> at the end of the file.
>>>>
>>>>
>>>> -- Jacob
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Mary Haley
>>>> <haley_at_ucar.edu>
>>>> Sent by:
>>>> To
>>>> ncl-talk-bounces@ Jacob.Klee_at_dom.com
>>>> ucar.edu
>>>> cc
>>>> 'ncl talk' <ncl-talk_at_ucar.edu>
>>>>
>>>> Subject
>>>> 07/03/2007 12:35 Re: [ncl-talk] memory error
>>>> PM question
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Hi Jacob,
>>>>
>>>> One of the places you can look if you start seeing memory allocation
>>>> problems is if you are creating a lot of arrays, but not freeing
>>>> them
>>>> up with "delete" after you are done with them. You already have
>>>> quite
>>>> a few "delete" calls in your script, but it might be helpful if you
>>>> could add more.
>>>>
>>>> I believe, yes, in this case you are running out of physical memory,
>>>> and deleting some arrays should help.
>>>>
>>>> The killer might be the triple do loop in which you are calling
>>>> addfile
>>>> inside the innermost loop, and then reading a bunch of variables off
>>>> the file.
>>>>
>>>> For example, you have code like this:
>>>>
>>>> SOILW00_01 = SOILW(0,:,:)
>>>> SOILW01_04 = SOILW(1,:,:)
>>>> SOILW04_10 = SOILW(2,:,:)
>>>> SOILW10_20 = SOILW(3,:,:)
>>>>
>>>> in which I think SOILW is not being used after this point, so you
>>>> could delete it.
>>>>
>>>> It might help us to know the intention of this program, so that
>>>> we could suggest some other ways to do this kind of heavy file i/o
>>>> and processing.
>>>>
>>>> Cheers,
>>>>
>>>> --Mary
>>>>
>>>>
>>>> On Tue, 3 Jul 2007 Jacob.Klee_at_dom.com wrote:
>>>>
>>>>
>>>>> Hello all,
>>>>>
>>>>> I have recived the error: fatal:NhlMalloc Failed:[errno=12]:Cannot
>>>>>
>>>> allocate
>>>>
>>>>> memory while running a memory intensive script. If my own memory
>>>>> has
>>>>>
>>> not
>>>
>>>>> yet failed, I remember this translates to running out of physical
>>>>>
>>> memory?
>>>
>>>>> I am running the prerelease of ....3.0 for CYGWIN on an XP machine
>>>>> with
>>>>>
>>>> 4gb
>>>>
>>>>> ram. I have included below the related files to my error.
>>>>>
>>>>> My questions are these: am I truly running out of memory? If so,
>>>>> any
>>>>> sugestions? If not, what is happening? (If anyone wants to run
>>>>> this, I
>>>>> would be happy to provide a handfull of the subseted NARR grib
>>>>> files I
>>>>>
>>> am
>>>
>>>>> using.)
>>>>>
>>>>>
>>>>>
>>>>> -- Jacob Klee
>>>>>
>>>>> (See attached file: ncl.exe.stackdump)(See attached file:
>>>>>
>>>> DCP_trimmed.log)
>>>>
>>>>> (See attached file: DomConvectParams_test.ncl)(See attached file:
>>>>> .hluresfile)
>>>>>
>>>>>
>>>>> -----------------------------------------
>>>>> CONFIDENTIALITY NOTICE: This electronic message contains
>>>>> information which may be legally confidential and/or privileged and
>>>>> does not in any case represent a firm ENERGY COMMODITY bid or offer
>>>>> relating thereto which binds the sender without an additional
>>>>> express written confirmation to that effect. The information is
>>>>> intended solely for the individual or entity named above and access
>>>>> by anyone else is unauthorized. If you are not the intended
>>>>> recipient, any disclosure, copying, distribution, or use of the
>>>>> contents of this information is prohibited and may be unlawful. If
>>>>> you have received this electronic transmission in error, please
>>>>> reply immediately to the sender that you have received the message
>>>>> in error, and delete it. Thank you.
>>>>>
>>>>>
>>>> _______________________________________________
>>>> ncl-talk mailing list
>>>> ncl-talk_at_ucar.edu
>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> ncl-talk mailing list
>>> ncl-talk_at_ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>
>>
>> --
>> ======================================================
>> Dennis J. Shea tel: 303-497-1361 |
>> P.O. Box 3000 fax: 303-497-1333 |
>> Climate Analysis Section |
>> Climate & Global Dynamics Div. |
>> National Center for Atmospheric Research |
>> Boulder, CO 80307 |
>> USA email: shea 'at' ucar.edu |
>> ======================================================
>>
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk_at_ucar.edu
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk_at_ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>

_______________________________________________
ncl-talk mailing list
ncl-talk_at_ucar.edu
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Mon Jul 09 2007 - 15:56:39 MDT

This archive was generated by hypermail 2.2.0 : Wed Jul 11 2007 - 08:55:54 MDT