Re: memory error question

Gladly.

I am 'brute-force' data mining... I am creating a csv file of 5 years
worth of data from the North American Regional Reanalysis dataset (01Mar
through 30Sep each year) and extracting / calculating numerous terms which
may have some contribution / explanitory value concerning disruptions to
our electric distribution network. (I curently have lightning data
standing in as a placeholder for our outtage data which has yet to be
compiled on the NARR grid.) I am going to feed the resulting file into
DTREG and start the fun analyzing the data, and hopefully in the end
producing a product which will further esitmate the potential impact of
upcoming weather to our system.

Also, the soil data is not involved in any calculations, but is writen out
at the end of the file.

-- Jacob

                                                                           
             Mary Haley
             <haley_at_ucar.edu>
             Sent by: To
             ncl-talk-bounces@ Jacob.Klee_at_dom.com
             ucar.edu cc
                                       'ncl talk' <ncl-talk_at_ucar.edu>
                                                                   Subject
             07/03/2007 12:35 Re: [ncl-talk] memory error
             PM question
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           

Hi Jacob,

One of the places you can look if you start seeing memory allocation
problems is if you are creating a lot of arrays, but not freeing them
up with "delete" after you are done with them. You already have quite
a few "delete" calls in your script, but it might be helpful if you
could add more.

I believe, yes, in this case you are running out of physical memory,
and deleting some arrays should help.

The killer might be the triple do loop in which you are calling addfile
inside the innermost loop, and then reading a bunch of variables off
the file.

For example, you have code like this:

   SOILW00_01 = SOILW(0,:,:)
   SOILW01_04 = SOILW(1,:,:)
   SOILW04_10 = SOILW(2,:,:)
   SOILW10_20 = SOILW(3,:,:)

in which I think SOILW is not being used after this point, so you
could delete it.

It might help us to know the intention of this program, so that
we could suggest some other ways to do this kind of heavy file i/o
and processing.

Cheers,

--Mary

On Tue, 3 Jul 2007 Jacob.Klee_at_dom.com wrote:

>
> Hello all,
>
> I have recived the error: fatal:NhlMalloc Failed:[errno=12]:Cannot
allocate
> memory while running a memory intensive script. If my own memory has not
> yet failed, I remember this translates to running out of physical memory?
>
> I am running the prerelease of ....3.0 for CYGWIN on an XP machine with
4gb
> ram. I have included below the related files to my error.
>
> My questions are these: am I truly running out of memory? If so, any
> sugestions? If not, what is happening? (If anyone wants to run this, I
> would be happy to provide a handfull of the subseted NARR grib files I am
> using.)
>
>
>
> -- Jacob Klee
>
> (See attached file: ncl.exe.stackdump)(See attached file:
DCP_trimmed.log)
> (See attached file: DomConvectParams_test.ncl)(See attached file:
> .hluresfile)
>
>
> -----------------------------------------
> CONFIDENTIALITY NOTICE: This electronic message contains
> information which may be legally confidential and/or privileged and
> does not in any case represent a firm ENERGY COMMODITY bid or offer
> relating thereto which binds the sender without an additional
> express written confirmation to that effect. The information is
> intended solely for the individual or entity named above and access
> by anyone else is unauthorized. If you are not the intended
> recipient, any disclosure, copying, distribution, or use of the
> contents of this information is prohibited and may be unlawful. If
> you have received this electronic transmission in error, please
> reply immediately to the sender that you have received the message
> in error, and delete it. Thank you.
>
_______________________________________________
ncl-talk mailing list
ncl-talk_at_ucar.edu
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Tue Jul 03 2007 - 11:11:39 MDT