Re: NCL termination due to memory problems

Hi Robert,

Without having access to your full script, I can only make a guess
about what might be wrong with your script. In the code snippet below,
you have multiple nested do loops, and in most interpreted languages
(like NCL), do loops can be a killer. Using nested do loops can
*really* slow a script down. I would recommend trying to see if you
can rewrite the code so that the nested do loops are not required.

Also, I'm not sure how you are writing out the netCDF files, but we
recommend that you use the "efficient" method, of which we have an
example at:

    http://www.ncl.ucar.edu/Applications/method_2.shtml

Also, it is not clear to me where the data is coming from that is
being written to the netCDF file, but you might consider using the
netCDF operators if you are operating on existing netCDF files. The
netCDF operators (http://nco.sourceforge.net/) are not part of the NCL
software package, but they are often used in conjunction with NCL and
can are fast.

Let me know if none of these suggestions are useful.

--Mary

On Thu, 27 Apr 2006, Robert F. Esswein wrote:

>
> I have a NCL script which is being prematurely terminated by
b> the system, apparently because of memory allocation problems.
> My efforts so far to find out why haven't gotten anywhere;
> maybe someone else has a clue. (I am not an experienced NCL user.)
>
> The structure of the script is more or less the following:
>
> begin
> ; some initialization
> do iday=0,nday-1
> ; some initialization
> do igrid=0,ngrid-1
> ; some initialization
> do istep=0,nstep-1
> ; some initialization
> ; Read one time step from input NetCDF file
> ; Interpolate to new grid
> print("writing output")
> system("date")
> ; Write to output NetCDF file. There is one output file per time step.> print("file written")
> end do
> end do
> end do
> end
>
> In the present case nday and ngrid == 1.
> The script dies at istep 20. (Some earlier versions ended at fewer
> steps. The script takes too long for me
> to investigate how reproducible this particular number is.)
> The command "dmesg" shows
> Out of Memory: Killed process 1892 (ncl).
> There are no messages output by NCL.
> There is no core dump.
>
> Monitoring with the "top" program shows a gradual increase in
> memory usage. Arrays used in the innermost loop have sizes determined
> by the grid sizes of interest, which are constant for the whole script.
> Arrays used outside the innermost loop do not amount to much.
>
> An earlier version, with grid sizes not much different,
> worked with no such problems.
> The major way the new version is different from the old one is that
> I am using my own procedure for interpolation instead of "linint2_Wrap".
>
> I do not understand how memory usage can be increasing with
> time, and why the innermost loop cannot complete. If it can complete
> several iterations, why can't it complete an indefinite number?
>
> The script exhibits additional strange behavior. For istep 9 and later
> output from system("date") does not appear even though output from the
> print statements just before and after it does appear.
>
> I am using version 4.2.0.a033 on a Linux system,
> Fedora Core release 3, kernel 2.6.9-1.667
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk_at_ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
_______________________________________________
ncl-talk mailing list
ncl-talk_at_ucar.edu
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Fri Apr 28 2006 - 08:52:15 MDT