Re: memory limit

Michael,

For arrays that large (1.6 and 2.2 Gbytes), I recommend processing
them in small sections sequentially, rather than having whole arrays
in memory. Will the nature of your processing allow this? What do
you mean by "I really need to work with all the data"?

Many math functions can be done by single steps along one dimension
such as lat, lon, or time. Others can be done in small bundles such
as one month or one year.

Some functions such as filters can be done with a sliding frame
across the time dimension, keeping only the time slabs within the
current frame in memory. The indexing can get a bit complicated,
but it is the price to pay for processing files and arrays larger
than available memory.

If not in conflict with the desired math functions, use the slowest
varying dimension in the file as the one to iterate. This is often
known as the "record dimension". This will speed up your program.

--Dave A.
CU/CIRES Climate Diagnostics Center (CDC)
NOAA/ESRL/PSD, Climate Analysis Branch (CAB)

Michael Notaro wrote:
> I am running the following on my Mac OS-X and am hitting a vm_allocate
> memory limit. The problem is that I create 2 large variables,
> tasflip(1200,360,720)
> and var(1560,360,720). My mac only allows me to create an array as large
> as (2200,360,720), so the code crashes. But I really need to work with
> all the data,
> so unless there is some trick to get around this problem, I am stuck.
> Any tips?
>
> Thanks,
> Michael
>
> ********************************
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) 6144
> file size (blocks, -f) unlimited
> max locked memory (kbytes, -l) unlimited
> max memory size (kbytes, -m) unlimited
> open files (-n) 256
> pipe size (512 bytes, -p) 1
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 100
> virtual memory (kbytes, -v) unlimited
> *********************************
>
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/shea_util.ncl"
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
> begin
>
> a = addfile("/Volumes/biggrape/bigmike/cru_ts_2_10.1901-2002.tmp.nc","r")
> tmp = a->tmp
> latit = a->lat
> longit = a->lon
>
> climo = clmMonTLL(tmp(828:1187,:,:)) ; 1970-1999 mean used for climo
>
> delete(tmp)
> delete(a)
>
> latit!0="lat"
> latit_at_units="degrees_north"
> latit&lat=latit
> longit!0="lon"
> longit_at_units="degrees_east"
> longit&lon=longit
>
> b =
> addfile("/Volumes/ftpstore2/IPCC/sresa2/atm/tas/ncar_ccsm3_0/run1/tas_A1.nc","r")
>
> lat = doubletofloat(b->lat)
> lon = doubletofloat(b->lon)
> time = doubletofloat(b->time)
> tas = b->tas
>
> c =
> addfile("/Volumes/ftpstore2/IPCC/20c3m/atm/tas/ncar_ccsm3_0/run1/tas_A1.20C3M_1.CCSM.atmm.1980-01_cat_1989-12.nc","r")
>
> d =
> addfile("/Volumes/ftpstore2/IPCC/20c3m/atm/tas/ncar_ccsm3_0/run1/tas_A1.20C3M_1.CCSM.atmm.1990-01_cat_1999-12.nc","r")
>
> modern1 = c->tas
> modern2 = d->tas
> mod1=clmMonTLL(modern1)
> mod2=clmMonTLL(modern2)
> modern = (mod1+mod2)/2.
>
> delete(modern1)
> delete(modern2)
> delete(mod1)
> delete(mod2)
> delete(c)
> delete(d)
>
> count=0
> do iy=0,99
> do im=0,11
> tas(count,:,:)=tas(count,:,:)-modern(im,:,:)
> count=count+1
> end do
> end do
>
> delete(modern)
>
> tasflip=lonFlip(tas)
> delete(tas)
> var=new((/1560,360,720/),float)
>
> .....etc...
>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk_at_ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
ncl-talk_at_ucar.edu
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Tue Feb 06 2007 - 13:28:38 MST