Re: potential addfiles problem

>From Dave Brown and tested by John Truesdale:

Before the the addfiles statement add

setfileoption("nc","SuppressClose",False)

Good luck
Melissa.Bukovsky_at_noaa.gov wrote:
> Hi,
>
> I'm pretty sure what I'm running into is not a memory problem as I've
> been watching it as the script runs and can read in all of the files and
> many variables from them if I split the process up.
>
> I don't get the add_connect error message, but many of these...
> "ncinquire: ncid 0: NetCDF: Not a valid ID". Here is the relevant part of
> the non-working script:
>
> fils = systemfunc("ls
> /rdata/mbukov/WRF/KF3090_cam_pdm/1991/wrfout_d02_1991-0*")
> f = addfiles(fils+".nc","r")
> ListSetType (f,"cat") ;concatenate
>
> p = f[:]->RAINC
>
> There is one wrfout file per 3 hours and using this method I can only
> read in about 4 months worth before the ncinquire error starts. If I
> specify one month only, it works fine.
>
>
> Here's the relevent part of the working version (I've made this one into
> a function)...
>
>
> ;field = name of variable to pull from file, all others are strings to
> do with the name
> ;and location of the files. yrs and mons = arrays of wanted years and months
>
> function
> getwrfout3d(field:string,yrs:string,mons:string,dom:string,dir:string,run:string)
>
> local nyrs,nmons,fils,flast,fnext,f,var
>
> begin
>
> nyrs = dimsizes(yrs)
> nmons = dimsizes(mons)
>
> do n = 0,nyrs-1 ;loop for years
> do m = 0,nmons-1 ;loop for months
> fils = systemfunc("ls
> "+dir+run+"/"+yrs(n)+"/wrfout_"+dom+"_"+yrs(n)+"-"+mons(m)+"*")
> fils = fils+".nc"
> f = addfiles(fils,"r")
> if(n.eq.0.and.m.eq.0)then
> var = addfiles_GetVar(f,fils,field)
> else
> vlast = var
> vnext = addfiles_GetVar(f,fils,field)
> delete(var)
> var = array_append_record(vlast,vnext,0)
> delete(vlast)
> delete(vnext)
> end if
> delete(fils)
> delete(f)
> end do
> end do
> return(var)
>
> end
>
> This version takes a little longer, but the only way to pull in all of
> the files was to add one month's worth at a time and append them. It
> also works with either version of addfiles while the previous script has
> a limit with either version of addfiles.
>
> Hope this helps. Let me know if you need anything else.
>
> Thanks,
>
> Melissa
>
>
>
>
> ----- Original Message -----
> From: John Truesdale <jet_at_ncar.ucar.edu>
> Date: Tuesday, May 13, 2008 11:37 am
> Subject: Re: potential addfiles problem
>
>> Hi Dennis and Mary:
>>
>> Previously Julie and I were running into problems when calling
>> addfiles with a large number of files. The error we were getting
>> was :
>>
>> add_connect: too many connects, cannot add another
>>
>> Mary discovered that this was a problem with the dap code inside ncl.
>> She made us an ncl.nodap version that worked for the numbers of
>> files that we were dealing with. I just tested that version though
>> and do run into a problem using more than 1402 files. The code
>> passes the addfiles statement but fails when I try to create a
>> variablewith:
>>
>> junk=fd[:]->OLR ;
>>
>> OLR is dimensioned [5612] x [256] x [512]
>> and fd[:] consists of 1407 files
>>
>> if I use
>>
>> junk=fd[:1402]->OLR
>>
>> it works fine
>>
>> but
>>
>> junk=fd[:1403]->OLR ; or junk=fd[:]->OLR ;
>>
>> fails with
>>
>> fatal:Either file (tmp_list_var_842) isn't defined or variable
>> (OLR) is not a variable in the file
>>
>> I don't think it is a memory thing because
>>
>> junk=fd[:1403]->OLR(:,0,0) ;
>>
>> also fails where the junk=fd[:1402]->OLR(:,:,:) passed and this var
>> is 256 x 512 larger ;
>>
>> Hope this helps the debug effort.
>>
>> jt
>>
>>
>> On Tue, 13 May 2008, Dennis Shea wrote:
>>
>>
>>> Mary
>>>
>>> Julie Caron and John Truesdale [ CGD; cc'd] have been running
>>>
>> into the
>>
>>> same problem with some WR\F related files.
>>>
>>> JC said u once built a special version of NCL.
>>>
>>> Are u in today ... or wfh?
>>>
>>> D
>>> Mary Haley wrote:
>>>
>>>> Hi Melissa,
>>>>
>>>> I was on vacation for a few days and just got back.
>>>>
>>>> The limit to the number of files you are able to read with
>>>>
>> "addfiles"> > could have to do with how much general memory you've
>> used in your code
>>
>>>> so far. I don't know what kind of limits you might run into
>>>> with addfiles, so I've CC'ed Dave Brown.
>>>>
>>>> Once you bring "addfiles_GetVar" into the picture, then you could
>>>> potentially be using up a lot more memory since you are now
>>>>
>> reading> > in actual data.
>>
>>>> Can you send us the script, and tell us how big the individual
>>>> files are?
>>>>
>>>> Also, what happens when you reach this limit? Does NCL quit
>>>>
>> with an
>>
>>>> error message, or does it hang, or something else entirely?
>>>>
>>>> --Mary
>>>>
>>>>
>>>> On Wed, 7 May 2008, Melissa Bukovsky wrote:
>>>>
>>>>
>>>>> Hi Mary,
>>>>>
>>>>> I seem to have ran into a limit with the addfiles or
>>>>>
>> addfiles_GetVar> > > functions. I haven't pinned down the number
>> exactly, but it seems to have
>>
>>>>> a limit to the number of files it can read in in one shot.
>>>>>
>> Somewhere> > > between 976 and 2928 files (I know that's a large
>> span, but that is how
>>
>>>>> the original script I wrote worked with all the files I
>>>>>
>> have). I've found
>>
>>>>> a work around to this since I need to read in many thousands
>>>>>
>> more than
>>
>>>>> this, but wanted to inquire about it anyway in case I hit it
>>>>>
>> again in the
>>
>>>>> future - i.e. my work around stops working for the number of
>>>>>
>> files I have.
>>
>>>>> Is there a set number in the addfiles functions, or am I
>>>>>
>> running into some
>>
>>>>> other problem?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Melissa Bukovsky
>>>>>
>>>>>
>>>>> ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
>>>>>
>> ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
>>
>>>>> Melissa Bukovsky
>>>>> GRA/Ph.D. Candidate
>>>>> OU School of Meteorology, NWC 5234
>>>>> mbukovsky_at_ou.edu
>>>>>
>>>>> "When one admits that nothing is certain one must, I think, also
>>>>> admit that some things are much more nearly certain than
>>>>>
>> others."> > > -Bertrand Russell
>>
>>>>> "Waiting for a sign to smash the silence with a brick of
>>>>> self control." -Armstrong
>>>>>
>>>>> "My opinions are my own and not representative of OU, CIMMS,
>>>>> NSSL, NOAA or any affiliates."
>>>>>
>>>>> =^..^=
>>>>>
>>>>>
>>>>>
>>>
>>>

-- 
======================================================
Dennis J. Shea                  tel: 303-497-1361    |
P.O. Box 3000                   fax: 303-497-1333    |
Climate Analysis Section                             |
Climate & Global Dynamics Div.                       |
National Center for Atmospheric Research             |
Boulder, CO  80307                                   |
USA                        email: shea 'at' ucar.edu |
======================================================
_______________________________________________
ncl-talk mailing list
ncl-talk_at_ucar.edu
http://mailman.ucar.edu/mailman/listinfo/ncl-talk