Re: memory issue

From: Mary Haley <haley_at_nyahnyahspammersnyahnyah>
Date: Mon Sep 30 2013 - 16:55:27 MDT

Hi Cheryl,

Generating lots of strings in an NCL script is known to be a serious memory hog. I'm not sure if that's the case, here, however.

How big are your two "do" loops (how big are "nday" and "ntim")? Also, how many lines are you generally writing to each ASCII file?

You might be able to use "write_table" instead of "write_matrix", in order to get around having to create arrays of formatted string output.

See:

http://www.ncl.ucar.edu/Document/Functions/Built-in/write_table.shtml

For example, instead of this code:

 lines = sprintf("%10.3f",lat)+sprintf("%10.3f",lon)+sprintf("%12.3f",tim)+sprintf("%10.3f",aod_l)+sprintf("%10.3f",aod_o)
 asciiwrite("file.txt",lines)

You could use this code, which writes the values directly to the file without creating a bunch of extra strings:

  write_table("file.txt","w",[/lat,lon,tim,aod_l,aod_o/],"%10.3f %10.3f %12.3f %10.3f %10.3f ")

--Mary

On Sep 27, 2013, at 8:34 AM, cheryl Ma <xiaoyancloud@gmail.com> wrote:

> Hi all,
>
> I monitored memory in use on my server and noticed it increased gradually when I run a ncl script. The ncl job stopped running when the memory is up to the limitation.
> Here is my script, Any hep on this issue would be very appreciated.
>
> Regards,
> Cheryl
>
> =================================
> begin
> nday = 31
> iday_start = 1
> do nd = 0, nday-1
> ;
> dir = "L2/"
> day = sprinti("%0.2i", nd+iday_start)
> ;
> fils = systemfunc("ls "+dir+"CER_SSF_Terra-FM1-MODIS_Edition3A_300301.200301"+day+"*.hdf")
>
> ntim = dimsizes(fils)
> do nt = 0,ntim-1
>
> lat = f->Colatitude_of_CERES_FOV_at_surface
> lon = f->Longitude_of_CERES_FOV_at_surface
> tim = f->Time_of_observation
> typ = f->Surface_type_index
> typ_perc = f->Surface_type_percent_coverage
> clayer=f->Clear_layer_overlap_percent_coverages
> aod_l = f->PSF_wtd_MOD04_corrected_optical_depth_land_0_550
> aod_o = f->PSF_wtd_MOD04_effective_optical_depth_average_ocean_0_550
>
> nfoot = dimsizes(lat(:))
>
> lines = sprinti("%10.0i",ntim)+sprinti("%10.0i",nfoot)
> asciiwrite("ASCII/200301/zNfoot_2003_01"+day+sprinti("%0.2i", nt+1)+".txt",lines)
>
> delete(lines)
> lines = sprintf("%10.3f",lat)+sprintf("%10.3f",lon)+sprintf("%12.3f",tim)+sprintf("%10.3f",aod_l)+sprintf("%10.3f",aod_o)
> asciiwrite("ASCII/200301/zAOD_2003_01"+day+sprinti("%0.2i", nt+1)+".txt",lines)
>
> opt = True
> opt@fout = "ASCII/200301/zTYP_2003_01"+day+sprinti("%0.2i", nt+1)+".txt"
> write_matrix (typ, "10i10", opt)
> ;
> opt = True
> opt@fout = "ASCII/200301/zTYC_2003_01"+day+sprinti("%0.2i", nt+1)+".txt"
> write_matrix (typ_perc, "10i10", opt)
>
> opt = True
> opt@fout = "ASCII/200301/zCLA_2003_01"+day+sprinti("%0.2i", nt+1)+".txt"
> write_matrix (clayer, "10f10.3", opt)
>
> delete(lines)
> delete(lat)
> delete(lon)
> delete(tim)
> delete(typ)
> delete(typ_perc)
> delete(clayer)
>
> end do
> ;
> delete(fils)
> delete(f)
> end do
> end
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Mon Sep 30 16:55:38 2013

This archive was generated by hypermail 2.1.8 : Tue Oct 01 2013 - 14:41:43 MDT