Re: NCL Wrapit + memory question

Jon,

Thanks for the clarification on your code. I didn't mean to imply that using Fortran was wrong, and you're right, this can be faster in some cases.

Back to why the "z" loop is growing in memory: I think the issue is with your rather large arrays. A 1460 x 361 x 720 float array is 1.4 GB in size. It's possible you are hitting a limit here with the amount of memory you're grabbing. Do you find that the script has to be killed in the same spot every time?

With the other scripts that you have that work, are they also dealing with arrays this large? What kind of system are you running on?

By the way, I see that you are writing the "time" variable as a character array. It is generally recommended not to do this, especially since you have the units as "Days since…" which implies that you have numeric values and not string values. We recommend using numeric values for time. If you really want a string time array, then you can always write this as a separate variable, but keep the "time" coordinate array as a numeric array.

What do your time values look like? It may be that you simply need to change:

     temp = str_get_field(str_get_field(f_list,6,"/"), 3, ".")
     time = stringtochar(temp)
     delete(temp)

to:

     time = tofloat(str_get_field(str_get_field(f_list,6,"/"), 3, "."))

and then instead of:

     dimNames = (/"time", "charlen", "lev", "lat", "lon"/)
     dimSizes = (/n_time, n_char, n_lev, n_lat, n_lon/)
     dimUnlim = (/ True, False, False, False, False/)
     filedimdef( ncf, dimNames, dimSizes, dimUnlim )

     filevardef( ncf, "time", "float", (/"time", "charlen"/) )

you would have:

     dimNames = (/"time", "lev", "lat", "lon"/)
     dimSizes = (/n_time, n_lev, n_lat, n_lon/)
     dimUnlim = (/ True, False, False, False/)
     filedimdef( ncf, dimNames, dimSizes, dimUnlim )

     filevardef( ncf, "time", "character", "time" )

--Mary

On Nov 18, 2013, at 10:02 AM, jonathan meyer <tallj03@hotmail.com> wrote:

> Hi Mary,
>
> I began using the fortran code because the loop was taking an exceptionally long time to crunch through, and fortran is
> MUCH faster.
>
>
> In a case like this, it might help if you could provide your datasets as well so we can try running the script
>
> This code is reading in a year of CFSR 6-hrly data (1460 x 37 x 361 x 720) for each 'yy' loop iteration, so it might be a little
> prohibitive to provide my dataset since it is so big.
>
>
>
> First, how are you determining that the memory is growing? From looking at the script, it does look like you are doing the right thing with the "delete" command. It also looks like you are doing a good job of defining the NetCDF file before you write to it.
>
> Using the 'top' command, I'm watching the memory grow with each 'z' interation. The script eventually is 'killed' when it
> reaches maximum memory.
>
>
> How big are your loops? That is, how many iterations are you doing with these two loops:
>
> For the loops, the yy loop is running from 1979-2009 (31 iterations) and the z loop is running for each vertical level (37).
> This gives variables for each 'z' iteration that are sized (1460 x 361 x 720).
>
>
>
>
> I tested the code this weekend by turning off the call to the fortran library and still saw the memory grow each 'z' iteration.
>
> Looking at the code, the only variable I could see that wasn't being deleted was the 'ncf' variable so I tried cutting the code down to write out a .nc file for each z iteration so I could delete the ncf variable after each 'z' iteration. With this, I was still seeing the memory grow. I'm rather confused as I have numerous other codes that are following similar paths but do not have a memory issue.
>
> Thanks for any advice.
> Jon
>
>
>
> > Subject: Re: NCL Wrapit + memory question
> > From: haley@ucar.edu
> > Date: Mon, 18 Nov 2013 09:48:33 -0700
> > CC: ncl-talk@ucar.edu
> > To: tallj03@hotmail.com
> >
> > Hi Jon,
> >
> > In a case like this, it might help if you could provide your datasets as well so we can try running the script.
> >
> > However, before we go that route, we can try some other things.
> >
> > First, how are you determining that the memory is growing? From looking at the script, it does look like you are doing the right thing with the "delete" command. It also looks like you are doing a good job of defining the NetCDF file before you write to it.
> >
> > The issue may be more with your nested do loops and not the Fortran code.
> >
> > How big are your loops? That is, how many iterations are you doing with these two loops:
> >
> > do yy = s_yy, e_yy
> >
> > do z = 0, n_lev-1
> >
> > ?
> >
> > Also, what version of NCL are you using?
> >
> > ncl -V
> >
> >
> > --Mary
> >
> > On Nov 15, 2013, at 2:15 PM, jonathan meyer <tallj03@hotmail.com> wrote:
> >
> > > Hi,
> > >
> > > I've been quite happy to learn how useful the wrapit functionality can be in NCL. While I've successfully gotten wrapit to work, I am seeing memory usage grow with each call to my subroutines.
> > >
> > > I've checked to make sure unused ncl variables are being deleted each time through my loop.
> > >
> > > Being rather green with the workings of fortran memory allocation, I'm wondering if there is something I'm missing in my fortran code that is allowing the arrays to remain in memory? I was under the impression that non-allocated memory in the subroutine would be removed at the 'end' command, but maybe this isn't the case for how I've coded mine.
> > >
> > > Attached is a code snippet of the NCL code for the loop the wrapit is being used, as well as the .f script containing my subroutines.
> > >
> > > As always, thanks in advance.
> > >
> > > Jon
> > > <Ageowinds.ncl><JDMlib.f>_______________________________________________
> > > ncl-talk mailing list
> > > List instructions, subscriber options, unsubscribe:
> > > http://mailman.ucar.edu/mailman/listinfo/ncl-talk
> >
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Mon Nov 18 11:32:28 2013