Re: memory issue

Please try write_table as I mentioned below.

I think the fact that you have 100,000 lines in each ASCII file could definitely cause a problem with string memory usage.

I did a quick timing test using the following short script with nday = 10 and ntim = 10, and saw a significant difference between using asciiwrite and write_table:

-------------------------------------------------------------------------------------------
  npts = 100000
  lat = fspan(-90,90,npts)
  lon = fspan(-180,180,npts)
  tim = fspan(0,500,npts)
  aod_l = random_uniform(-100,100,npts)
  aod_o = random_uniform(0,200,npts)

  total_ascii = 0. ; keep track of timings
  total_table = 0.

  nday = 2
  ntim = 2
  do i=0,nday-1
    do j=0,ntim-1
;---Using asciiwrite
      start_ascii = get_cpu_time()
      lines = sprintf("%10.3f",lat)+sprintf("%10.3f",lon)+sprintf("%12.3f",tim)+\
              sprintf("%10.3f",aod_l)+sprintf("%10.3f",aod_o)
      asciiwrite("file.txt",lines)

      end_ascii = get_cpu_time()
      total_ascii = total_ascii+(end_ascii-start_ascii)

;---Using write_table
      start_table = get_cpu_time()
      write_table("file2.txt","w",[/lat,lon,tim,aod_l,aod_o/],"%10.3f %10.3f %12.3f %10.3f %10.3f ")
      end_table = get_cpu_time()
      total_table = total_table+(end_table-start_table)
    end do
  end do

  print("asciiwrite: " + total_ascii + " total CPU seconds.")
  print("write_table: " + total_table + " total CPU seconds.")
-------------------------------------------------------------------------------------------

(0) asciiwrite: 14.3604 total CPU seconds.
(0) write_table: 2.86116 total CPU seconds.

You can see with just nday = ntim = 2, there's a difference in the two methods.

Also, since you have 100,000 lines, and each line has at least 5 columns, this is 5 * 100,000 strings that have to get created when you call sprintf and sprinti. Furthermore, you are doing this for 31*24 timesteps. This is going to get slower and slower as you go.

May I ask why you need to create ASCII files? Is there a reason you can't use NetCDF or some other more efficient binary format?

--Mary

On Oct 1, 2013, at 7:34 AM, cheryl Ma <xiaoyancloud@gmail.com> wrote:

> Mary, thanks for taking your time.
>
> nday = 31 and ntim=24, The lines writing to each ASCII file up to 100000.
>
>
>
> On Mon, Sep 30, 2013 at 6:55 PM, Mary Haley <haley@ucar.edu> wrote:
> Hi Cheryl,
>
> Generating lots of strings in an NCL script is known to be a serious memory hog. I'm not sure if that's the case, here, however.
>
> How big are your two "do" loops (how big are "nday" and "ntim")? Also, how many lines are you generally writing to each ASCII file?
>
> You might be able to use "write_table" instead of "write_matrix", in order to get around having to create arrays of formatted string output.
>
> See:
>
> http://www.ncl.ucar.edu/Document/Functions/Built-in/write_table.shtml
>
>
> For example, instead of this code:
>
> lines = sprintf("%10.3f",lat)+sprintf("%10.3f",lon)+sprintf("%12.3f",tim)+sprintf("%10.3f",aod_l)+sprintf("%10.3f",aod_o)
> asciiwrite("file.txt",lines)
>
> You could use this code, which writes the values directly to the file without creating a bunch of extra strings:
>
> write_table("file.txt","w",[/lat,lon,tim,aod_l,aod_o/],"%10.3f %10.3f %12.3f %10.3f %10.3f ")
>
>
> --Mary
>
>
>
> On Sep 27, 2013, at 8:34 AM, cheryl Ma <xiaoyancloud@gmail.com> wrote:
>
> > Hi all,
> >
> > I monitored memory in use on my server and noticed it increased gradually when I run a ncl script. The ncl job stopped running when the memory is up to the limitation.
> > Here is my script, Any hep on this issue would be very appreciated.
> >
> > Regards,
> > Cheryl
> >
> > =================================
> > begin
> > nday = 31
> > iday_start = 1
> > do nd = 0, nday-1
> > ;
> > dir = "L2/"
> > day = sprinti("%0.2i", nd+iday_start)
> > ;
> > fils = systemfunc("ls "+dir+"CER_SSF_Terra-FM1-MODIS_Edition3A_300301.200301"+day+"*.hdf")
> >
> > ntim = dimsizes(fils)
> > do nt = 0,ntim-1
> >
> > lat = f->Colatitude_of_CERES_FOV_at_surface
> > lon = f->Longitude_of_CERES_FOV_at_surface
> > tim = f->Time_of_observation
> > typ = f->Surface_type_index
> > typ_perc = f->Surface_type_percent_coverage
> > clayer=f->Clear_layer_overlap_percent_coverages
> > aod_l = f->PSF_wtd_MOD04_corrected_optical_depth_land_0_550
> > aod_o = f->PSF_wtd_MOD04_effective_optical_depth_average_ocean_0_550
> >
> > nfoot = dimsizes(lat(:))
> >
> > lines = sprinti("%10.0i",ntim)+sprinti("%10.0i",nfoot)
> > asciiwrite("ASCII/200301/zNfoot_2003_01"+day+sprinti("%0.2i", nt+1)+".txt",lines)
> >
> > delete(lines)
> > lines = sprintf("%10.3f",lat)+sprintf("%10.3f",lon)+sprintf("%12.3f",tim)+sprintf("%10.3f",aod_l)+sprintf("%10.3f",aod_o)
> > asciiwrite("ASCII/200301/zAOD_2003_01"+day+sprinti("%0.2i", nt+1)+".txt",lines)
> >
> > opt = True
> > opt@fout = "ASCII/200301/zTYP_2003_01"+day+sprinti("%0.2i", nt+1)+".txt"
> > write_matrix (typ, "10i10", opt)
> > ;
> > opt = True
> > opt@fout = "ASCII/200301/zTYC_2003_01"+day+sprinti("%0.2i", nt+1)+".txt"
> > write_matrix (typ_perc, "10i10", opt)
> >
> > opt = True
> > opt@fout = "ASCII/200301/zCLA_2003_01"+day+sprinti("%0.2i", nt+1)+".txt"
> > write_matrix (clayer, "10f10.3", opt)
> >
> > delete(lines)
> > delete(lat)
> > delete(lon)
> > delete(tim)
> > delete(typ)
> > delete(typ_perc)
> > delete(clayer)
> >
> > end do
> > ;
> > delete(fils)
> > delete(f)
> > end do
> > end
> >
> > _______________________________________________
> > ncl-talk mailing list
> > List instructions, subscriber options, unsubscribe:
> > http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Tue Oct 1 09:33:47 2013