Re: NclMalloc error

All,

We are seeing a lot of large array problems recently among NCL users.
There is a general approach that works very well for most
applications. Process your data in slices along one dimension,
instead of trying to read the entire data set into memory all at once.
 This can reduce the total memory requirement by orders of magnitude.

This usually involves adding just a single outer loop over one
dimension. The dimension must be selected carefully to not interfere
with the desired computation. For example, if computing spatial area
averages, you would not want to slice along the X and Y dimensions,
but slicing along the time or level dimensions would probably be fine.
 Temporal filters can easily be sliced along X or Y, but not time.

Adding the loop and doing the subscripting correctly does make the
code a little more complicated. You will need to learn how to
allocate memory variables and file variables by their final dimension
sizes, and how to subscript properly to fill these arrays one slice at
a time. Also, keep as much as possible outside the loop, such as
opening files and creating plots.

Many data processing tasks for large data sets can be conquered by
these methods without requiring huge computer memory. Try this as an
alternative to seeking ever larger physical memory size.

--Dave

On Thu, Dec 20, 2012 at 9:13 AM, Panday, Prajjwal <ppanday@clarku.edu> wrote:
> Hello all,
>
> I am trying to compute monthly anomalies for the CRU data(~2.5GB) using NCL
> version 6.1.0. I am using NCL installed through cygwin. However, I get the
> error
> "fatal:NclMalloc Failed: [errno=12]
> fatal: Error allocating -90600955556656555 bytes to read variable ..."
>
> I realize this is the NCL memory limit error but I do need to work on the
> entire dataset. Is there any workaround this issue?
>
> Regards,
> Prajjwal
>
> Code snippet
> -----------------------------------------------------------------------------------------
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
> load "$NCARG_ROOT/lib/ncarg/nclscripts/esmf/ESMF_regridding.ncl"
>
>
> ;*********************************************************************
> ; Read in input NetCDF files
> ;*********************************************************************
> dir2 = "C:/Data/Documents/IBIS/reanalysis_data/cru/"
> file2 = "cru_pre.nc"
> cru_prec = addfile(dir2+file2,"r")
> setfileoption("nc","SuppressClose",False)
> cru_mon = cru_prec->pre(:,:,:)
> printVarSummary(cru_mon)
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Thu Dec 20 11:20:10 2012