Re: Memory allocation

From: Wei Huang <huangwei_at_nyahnyahspammersnyahnyah>
Date: Thu Oct 15 2009 - 09:56:04 MDT

Anandhi,

I do not a solution for this problem.
But I guess you can by-pass this problem
by processing your files one at a time.
Then you will have 195 ascii files,
and you can cat them together with a Unix command.

You may still be able to process all files at once
at Unix level, with Unix script (or Unix like script).

Hope this can get your work going.

Wei Huang
huangwei@ucar.edu
VETS/CISL
National Center for Atmospheric Research
P.O. Box 3000 (1850 Table Mesa Dr.)
Boulder, CO 80307-3000 USA
(303) 497-8924

On Oct 15, 2009, at 9:20 AM, Anandhi Aavudai wrote:

> Hi NCL users,
>
> I am repeating my earlier query from "ncl-talk Digest, Vol 71, Issue
> 36" again as I am not sure the reason for not getting any response.
>
>
> I have a script which reads about 195 .nc files one at a time in a
> do loop. Then each file is regridded and a section is written into a
> ascii file using 2 other do loops. To my knowledge all the memory
> allocated for each file in reading, regridding & writing to ascii
> file is released by using delete statements. In am using ncl 5.1.1
> in Cywin. my doubts are
>
> 1. when I track the time required in manipulating each of the 195 nc
> files, Even when all the files are of equal size, why do I find the
> time required for each file manipulation is almost doubled as more &
> more files are read ?
> Eg. Given that the files are of same size, it took 425 sec for the
> first file, it takes around 900 secs for the second file, 1824 secs
> for third file, & 3575 secs for the forth etc..
>
> 2. on reading the previous posting in ncl-talk archives, I
> understand creating unique strings in a loop consumes a lot of
> memory. But I am deleting the array/variables having the string at
> the end. Why is my program still consuming more memory? Finally the
> program stops with error "fatal:NhlMalloc Failed:[errno=12]" . where
> is the memory taken up in the program?
>
> 3. in Matlab the current directory lets me know which are the
> variables/arrays are still defined after running a program. Is there
> something similar or ohter commands in NCL which will let me know
> which variable/arrays are taking up memory after completing a loop?
> this could help me know why time is doubled.
>
> 4. How do I upload a couple of my nc files?
>
> Thanks in advance for the help.
>
> regards
> Anandhi
>
> ;;;My program-------------------
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
> begin
> dir = "F:/IPCC_data_copy/sresa1b_atm_da/pr"
> dblquote = integertochar(34)
> fili = systemfunc("find " + dir + "/* -type f -name " + dblquote +
> "*.nc" + dblquote + " -
> print") ;=========================================;
> do nf=0,nfili-1 ; loop to read each file
> wcStrt= systemfunc("date") ; start of cal. time for
> this prog
> in = addfile (fili(nf), "r") ; reads a file
> orig=in->pr
> time = in->time
> lat = in->lat
> lon = in->lon({0:360})
>
> ;====Regridding----
> newlat=fspan(-90.,90,73);
> newlon=fspan(0.,360,145);
> newlat!0 = "lat"
> newlat@units = "degress_north"
> newlat@long_name = "latitude"
> newlat&lat = newlat
> newlon!0 = "lon"
> newlon@units = "degress_east"
> newlon@long_name = "longitude"
> newlon&lon = newlon
> dout=linint2_Wrap(lon,lat,orig,True,newlon,newlat,0)
> ;======Output file name
> fout=new(3,"string") ; defing fout ass an array of strings
> fout(0)="G:/IPCC_data_copy/sresa1b_atm_da_all_regrid/pr" ;
> adding a folder name
> fex=stringtochar(fili(nf));
> fout(1)=chartostring(fex(35:dimsizes(fex)-5)) ; extracting a
> portion of the file name from input file
> fout(2)="_NHe_out.txt"
> fNameO=str_concat(fout)
> print(fNameO)
>
> nnelat=dimsizes(newlat)
> nnelon=dimsizes(newlon)
> ntim=dimsizes(time)
> data=new(19*15*ntim,"string"); pre allocate
> Time=sprintf("%7.1f ",in&time) ; do once: always the same
> nStrt = 0
> nLast = ntim-1
> do nl=44,62 ; extracts lat 22.5N to 65N -- 19 latiude values
> do ml = 108,122 ; extracts 270 to 305 ---15 longitudes
> LatLon = sprintf("%7.1f ",newlat(nl)) +
> sprintf("%6.1f" , newlon(ml)) + sprintf("%6.1f" , 360-newlon(ml))
> data(nStrt:nLast) = Time + LatLon +
> sprintf("%25.2f",dout(:,nl,ml)*86400)
> nStrt = nStrt + ntim
> nLast = nLast + ntim
> end do
> end do
> asciiwrite(fNameO,data)
> wallClockElapseTime(wcStrt, "Ascii: "+fili(nf) ,0) ; end & print
> timelapsed since the clock start time loop
> delete(in)
> delete(dout)
> delete(lat)
> delete(lon)
> delete(newlat)
> delete(newlon)
> delete(nnelat)
> delete(nnelon)
> delete(ntim)
> delete(fNameO)
> delete(data)
> delete(fout)
> delete(orig)
> delete(time)
> delete(Time)
> delete(wcStrt)
> delete(LatLon)
> end do ; nf loop
> end
>
>
> -----------------------------
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Thu, 15 Oct 2009 09:56:04 -0600

This archive was generated by hypermail 2.1.8 : Thu Oct 22 2009 - 12:22:52 MDT