Re: Memory allocation

From: Dennis Shea <shea_at_nyahnyahspammersnyahnyah>
Date: Thu Oct 15 2009 - 10:12:33 MDT

You can upload 3 or 4 files.
Please see:

http://www.ncl.ucar.edu/

Under "Support", click "Report Bugs"

There is an ftp site to which you can upload a few sample files.
As noted, you must send us the files names.
Just send them directly to me ... less ncl-talk traffic.

====
I am sure you have your reasons for creating ASCII.
It is the least efficient format for storage.

I think a better procedure for writing ascii is

http://www.ncl.ucar.edu/Document/Functions/Built-in/write_matrix.shtml

   opt = True
   opt@fout = "FOO"
   write_matrix (x, fmtx, opt)

=====

I'd suggest making netCDF [perhaps just one big file].
Then, if ascii is needed, read each variable
and write to ascii files via

---
FYI only
There is an example of 'cat'-ing using systemfunc
http://www.ncl.ucar.edu/Applications/read_ascii.shtml
see example 6
Anandhi Aavudai wrote:
> Hi NCL users,
> 
> I am repeating my earlier query from "ncl-talk Digest, Vol 71, Issue 36" again as I am not sure the reason for not getting any response.
> 
> 
> I have a script which reads about 195 .nc files one at a time in a do loop. Then each file is regridded and a section is written into a ascii file using 2 other do loops. To my knowledge all the memory allocated for each file in reading, regridding & writing to ascii file is released by using delete statements. In am using ncl 5.1.1 in Cywin. my doubts are
> 
> 1. when I track the time required in manipulating each of the 195 nc files, Even when all the files are of equal size, why do I find the time required for each file manipulation is almost doubled as more & more files are read ?
> Eg. Given that the files are of same size, it took 425 sec for the first file, it takes around 900 secs for the second file, 1824 secs for third file, & 3575 secs for the forth etc..
> 
> 2. on reading the previous posting in ncl-talk archives, I understand creating unique strings in a loop consumes a lot of memory. But I am deleting the array/variables having the string at the end. Why is my program still consuming more memory? Finally the program stops with error "fatal:NhlMalloc Failed:[errno=12]" . where is the memory taken up in the program?
> 
> 3. in Matlab the current directory lets me know which are the variables/arrays are still defined after running a program. Is there something similar or ohter commands in NCL which will let me know which variable/arrays are taking up memory after completing a loop? this could help me know why time is doubled.
> 
> 4. How do I upload a couple of my nc files?
> 
> Thanks in advance for the help.
> 
> regards
> Anandhi
> 
> ;;;My program-------------------
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
> begin
> dir = "F:/IPCC_data_copy/sresa1b_atm_da/pr"
> dblquote = integertochar(34)
> fili = systemfunc("find " + dir + "/* -type f -name " + dblquote + "*.nc" + dblquote + " -print") ;=========================================;
> do nf=0,nfili-1      ; loop to read each file
>           wcStrt= systemfunc("date")       ; start of cal. time for this prog
>           in = addfile (fili(nf), "r") ; reads a file
>           orig=in->pr
>           time = in->time
>           lat = in->lat
>           lon = in->lon({0:360})
> 
>         ;====Regridding----
>          newlat=fspan(-90.,90,73);
>          newlon=fspan(0.,360,145);
>          newlat!0 = "lat"
>          newlat@units = "degress_north"
>          newlat@long_name = "latitude"
>          newlat&lat = newlat
>          newlon!0 = "lon"
>          newlon@units = "degress_east"
>          newlon@long_name = "longitude"
>          newlon&lon = newlon
>         dout=linint2_Wrap(lon,lat,orig,True,newlon,newlat,0)
>         ;======Output file name
>         fout=new(3,"string") ; defing fout ass an array of strings
>        fout(0)="G:/IPCC_data_copy/sresa1b_atm_da_all_regrid/pr" ; adding a folder name
>        fex=stringtochar(fili(nf));
>        fout(1)=chartostring(fex(35:dimsizes(fex)-5)) ; extracting a portion of the file name from input file
>        fout(2)="_NHe_out.txt"
>        fNameO=str_concat(fout)
>        print(fNameO)
> 
>        nnelat=dimsizes(newlat)
>        nnelon=dimsizes(newlon)
>        ntim=dimsizes(time)
>        data=new(19*15*ntim,"string"); pre allocate
>        Time=sprintf("%7.1f ",in&time) ; do once: always the same
>        nStrt = 0
>        nLast = ntim-1
>       do nl=44,62     ; extracts  lat 22.5N to 65N -- 19 latiude values
>                do ml = 108,122   ; extracts 270 to 305 ---15 longitudes
>                        LatLon = sprintf("%7.1f ",newlat(nl)) + sprintf("%6.1f" , newlon(ml)) + sprintf("%6.1f" , 360-newlon(ml))
>                        data(nStrt:nLast) = Time + LatLon + sprintf("%25.2f",dout(:,nl,ml)*86400)
>                        nStrt = nStrt + ntim
>                        nLast = nLast + ntim
>                 end do
>        end do
>       asciiwrite(fNameO,data)
>      wallClockElapseTime(wcStrt, "Ascii: "+fili(nf) ,0) ; end & print timelapsed since the clock start time loop
>      delete(in)
>      delete(dout)
>      delete(lat)
>      delete(lon)
>      delete(newlat)
>      delete(newlon)
>      delete(nnelat)
>      delete(nnelon)
>      delete(ntim)
>      delete(fNameO)
>      delete(data)
>      delete(fout)
>      delete(orig)
>      delete(time)
>      delete(Time)
>      delete(wcStrt)
>      delete(LatLon)
> end do ; nf loop
> end
> 
> 
> -----------------------------
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Thu, 15 Oct 2009 10:12:33 -0600

This archive was generated by hypermail 2.1.8 : Thu Oct 22 2009 - 12:22:52 MDT