Re: NCL is getting slower when I call "user defined function" many times

From: Joowan Kim <joowan.k_at_nyahnyahspammersnyahnyah>
Date: Wed Apr 24 2013 - 17:22:18 MDT

Dear Dave

Thanks for the clear explanation. It helps a lot (and gave me more
confidence on NCL for building my own analysis function set).
I guess it will be included in the next lease.
At this moment I detour the problem by modifying my coding. I can wait
until next release comes.

Thanks so much!

Joowan

On Wed, Apr 24, 2013 at 5:16 PM, David Brown <dbrown@ucar.edu> wrote:

> Hi Joowan,
>
> FYI, I have fixed this problem and your script can now run to completion
> with essentially no increase in time for iterations of the loop from
> beginning to end. For the record, it has nothing
> to do with whether a function is user-defined or not. The code was losing
> track of small objects holding scalar values in 2 situations: 1) when
> taking the short-circuit path for .and. and .or. logical expressions, and
> 2) when creating the _FillValue attribute for certain local variables.
>
> Let us know if you would like to get an updated binary.
> -dave
>
>
> On Apr 23, 2013, at 7:49 PM, Joowan Kim <joowan.k@gmail.com> wrote:
>
> Dear David
>
> Thanks for your effort and kind explanation.
> I appreciate it!
>
> Regards,
>
> Joowan
>
>
>
> On 23 April 2013 14:15, David Brown <dbrown@ucar.edu> wrote:
>
>> Hi Joowan,
>>
>> We have been able to duplicate the issue you raise here. The problem is
>> that there are some small objects that are created in great numbers but do
>> not get removed as the do loops progress. They do take up memory but the
>> real cause of the slowdown is that access to the object table slows as the
>> table gets really large. I will try to get to the bottom of this problem as
>> soon as I can.
>> -dave
>>
>> On Apr 22, 2013, at 10:31 AM, Joowan Kim <joowan.k@gmail.com> wrote:
>>
>> Dear NCL users
>>
>> I'm calculating the tropopause for temperature profiles using NCL's "do
>> loop" and a user-defined function.
>> The user-defined function works for a single temperature profile, and do
>> loop call the function for each temperature profiles.
>> There is an exactly same calculation ("just calling of a function I
>> defined") in the do loop, so each step is expected to take a same time.
>> However the do loop takes longer and longer time as it progresses.
>>
>> Here is the example
>> (elapsed time (time1-time0) for each step keeps increasing)
>> [joowan@rossby ttl_joowan]$ ncl *main.ncl*
>> Copyright (C) 1995-2011 - All Rights Reserved
>> University Corporation for Atmospheric Research
>> NCAR Command Language Version 6.0.0
>> The use of this software is governed by a License Agreement.
>> See http://www.ncl.ucar.edu/ for more details.
>> (0) loop i = 1 elapsed time: 4
>> (0) loop i = 2 elapsed time: 5
>> (0) loop i = 3 elapsed time: 7
>> (0) loop i = 4 elapsed time: 9
>> (0) loop i = 5 elapsed time: 10
>> (0) loop i = 6 elapsed time: 11
>> (0) loop i = 7 elapsed time: 13
>> (0) loop i = 8 elapsed time: 14
>> (0) loop i = 9 elapsed time: 15
>> (0) loop i = 10 elapsed time: 17
>> (0) loop i = 11 elapsed time: 19
>> (0) loop i = 12 elapsed time: 20
>> ...
>> ...
>> (0) loop i = 31 elapsed time: 52
>> (0) loop i = 32 elapsed time: 54
>>
>> I tested this with the newest version 6.1.2, but it has the same problem.
>> It is very strange because I never met this problem with NCL before.
>> One thing I found is memory use keeps increase until unreasonable size. I
>> guess "user defined function" doesn't return memory, so lots of calling it
>> takes bigger memory and longer time. (If I directly do the calculation in
>> the code, not using a user-defined function, that slow down does NOT
>> happen.) But it is just my crude guess...
>> It is a serious problem for me, because the do loop goes up to 3000 for
>> model analysis, and the computation time grows exponentially.
>> Is there any Known issue for this problem or treatment?
>> Any comment will be appreciated.
>>
>> Thanks so much
>>
>> Best!
>> Just in case, I attached my script and data
>>
>>
>> Joowan
>>
>> P.S.
>> I'm using
>> Linux rossby.meteo.mcgill.ca 2.6.34.9-69.fc13.x86_64 #1 SMP Tue May 3
>> 09:23:03 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
>> GCC version is "gcc version 4.4.5 20101112 (Red Hat 4.4.5-2) (GCC)"
>>
>>
>> --
>> Joowan Kim
>> Atmospheric and Oceanic Sciences
>> McGill University
>> e-mail : joowan.kim@mail.mcgill.ca
>> joowan.k@gmail.com
>> <main.ncl><ttl_joowan.ncl><tprof_ERAI.dat>
>> _______________________________________________
>>
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>>
>
>
> --
> Joowan Kim
> Atmospheric and Oceanic Sciences
> McGill University
> e-mail : joowan.kim@mail.mcgill.ca
> joowan.k@gmail.com
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
>

-- 
Joowan Kim
Atmospheric and Oceanic Sciences
McGill University
e-mail : joowan.kim@mail.mcgill.ca
                joowan.k@gmail.com

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Wed Apr 24 17:22:59 2013

This archive was generated by hypermail 2.1.8 : Fri Apr 26 2013 - 17:10:25 MDT