Re: addfiles

From: Gus Correa <gus_at_nyahnyahspammersnyahnyah>
Date: Thu Apr 15 2010 - 17:20:44 MDT

Hi Yang Yang

The 2008 posting/suggestion was from Dennis Shea, not me.
So, thanks go to him.

**

Assuming double precision data:

12*4*323*322*58*8 = 2,316,421,632 bytes ~= 2.3GB

Your /proc/meminfo reports 7.8GB, which is more.

However, I think you may be using twice as much, 4.6GB,
because you have f1[:]->soil_moisture(:,:,:,:)
*and* smm. Maybe more than that, if you have other big
variables in your script, or other programs running.

You may have run out of memory (the "malloc" errors).
You can monitor the memory being used with "top",
on a separate window, while your NCL script is running.

Are there other programs running along with NCL,
that may be taking memory?
Can you turn them off?

Do you have other big variables in your NCL script?
Can you delete them before getting to this point?

Any syntax that can be simplified to avoid the use of
(big) temporary variables?
E.g., can you avoid "smm", and use f1[:]->soil_moisture(:,:,:,:)
directly in your calculation?

Can you perhaps loop over slices of the data,
say, on the file indes, or the time dimension,
or soil level, to do your calculation?

Gus Correa

Yang Yang wrote:
> Dear Dave, Gus, and Mary:
>
> Thank you very much for your help!
>
> With the suggestion from Dave and Gus, the problem is solved. The version of NCL I used is 5.0.0. So we need to update it. However, I encountered another problem for a ncl statement as:
>
> smm = f1[:]->soil_moisture(:,:,:,:) with the following error info.
>
> fatal:NclMalloc Failed:[errno=12]
> fatal:_NclBuildArray : Memory allocation failure
> fatal:Execute: Error occurred at or near line 34 in file read_plot_1.ncl
>
> If I make a change as
>
> smm = f1[:]->soil_moisture(:,0,:,:)
>
> It works.
>
> The system of our workstation is linux with the following memory size:
>
> $ cat /proc/meminfo |grep -i memtotal
> MemTotal: 7886772 kB
>
> The dim of soil_moisture is (12,4,323,322), and that of f1 is (58)
>
> The problem occurred is because the memory is not enough? Can I make other setting so that the problem can be solved? how?
>
> Thank you!
>
> Regards
> Ed
>
> --- On Fri, 4/16/10, David Brown <dbrown@ucar.edu> wrote:
>
>> From: David Brown <dbrown@ucar.edu>
>> Subject: Re: addfiles
>> To: "Yang Yang" <yang123yang@yahoo.com>
>> Cc: "ncl-talk forum" <ncl-talk@ucar.edu>
>> Date: Friday, April 16, 2010, 12:55 AM
>> Hi Ed,
>>
>> It looks like you are using an opendap-enabled version of
>> NCL. If it a version of NCL earlier than 5.2.0 then
>> all NetCDF file access is through the OPeNDAP NetCDF
>> client library which limits the number of simultaneous
>> connections (open files) to 64. Given that there are a small
>> number of connections that are pre-allocated and not
>> actually available, it is not surprising that you are
>> running out of descriptors. The solution is to use the
>> setfileoption procedure:
>>
>> setfileoption("nc", "SuppressClose", False)
>>
>> prior to your addfiles call. See the last paragraph of the
>> addfiles Description section at http://www.ncl.ucar.edu/Document/Functions/Built-in/addfiles.shtml.
>>
>> -dave
>>
>>
>> On Apr 15, 2010, at 10:11 AM, Mary Haley wrote:
>>
>>> Hi Ed,
>>>
>>> I don't think opening 58 files should be a problem,
>> unless they are
>>> large and you run
>>> out of memory. I created a dummy set of 59 files and
>> read them all
>>> fine with a
>>> single call to "addfiles".
>>>
>>> What version of NCL are you running (ncl -V)?
>>>
>>> Are the files on the web somewhere that I can access
>> them? You can email
>>> me offline with this information.
>>>
>>> If you can't provide the files, then I will need more
>> information,
>>> like how you
>>> are trying to open the files, (using "cat" or "join"),
>> whether you
>>> are getting
>>> any other error messages, etc.
>>>
>>> Thanks,
>>>
>>> --Mary
>>>
>>>
>>>
>>> On Apr 14, 2010, at 9:55 PM, Yang Yang wrote:
>>>
>>>> Hello:
>>>>
>>>> I have 58 files and want to use addfiles to add
>> them together.
>>>> However, NCL shows the follow information:
>>>>
>>>> add_connect: too many connects, cannot add
>> another
>>>> ncinquire: ncid -1: NetCDF: Not a valid ID
>>>> add_connect: too many connects, cannot add
>> another
>>>> ncinquire: ncid -1: NetCDF: Not a valid ID
>>>>
>>>> Can you tell me at most how many files I can use
>> addfiles to add
>>>> them together? Thanks!
>>>>
>>>> Regards
>>>> Ed
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> ncl-talk mailing list
>>>> List instructions, subscriber options,
>> unsubscribe:
>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>> _______________________________________________
>>> ncl-talk mailing list
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Thu Apr 15 17:21:03 2010

This archive was generated by hypermail 2.1.8 : Fri Apr 16 2010 - 16:21:29 MDT