NCL can't perform one simple average with large file? Memory Issue

From: DJ Rasmussen <dj.rasmussen_at_nyahnyahspammersnyahnyah>
Date: Sat, 03 Oct 2009 18:27:48 -0500

NCL Community,

I have 14 u and v wind component CRCM files that I would
like to average. I have been trying to create a script
that reads the files in and performs one giant average of
all of the data, but I am running into memory errors. Even
averaging just one file alone gives an error with this

  Error: fatal:NclMalloc Failed:[errno=12]

load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"


; read in multiple netCDF files
diri = "/home/rasmussen/NARCCAP/netCDF/RCM3/" ; directory
where files reside
fils = systemfunc("ls "+diri+"uas_CRCM_*.nc")
nfil = dimsizes(fils)

;*************************Get Variables**********
  f = addfiles (fils, "r")

    UAS = f[:]->uas ; (time,yc,xc)


;*************************Calculate Average******

    WSPD_AVG = dim_avg_n(UAS,0) ; (lat,lon)
    WSPD_AVG_at_long_name = "mean wind speed"


; output ascii

  fName = "RCM3AVG.txt"
  system("/bin/rm -f "+fName) ; remove if file is present
  asciiwrite (fName , WSPD_AVG)


The basic script can read smaller files with less time
steps just fine (8760 steps), but it gives an error with
the files that have 14600 time steps. (3hrly over 5 years)

Variable: UAS
Type: float
Total Size: 940240000 bytes
             235060000 values
Number of Dimensions: 3
Dimensions and sizes: [time | 14600] x [yc | 115] x [xc |
             time: [8395.125..10220]
             yc: [ 0..5700000]
             xc: [ 0..6950000]
Number Of Attributes: 13
   units : m s-1
   type : instantaneaous
   long_name : Zonal Surface Wind Speed
   standard_name : eastward_wind
   missing_value : 1e+20
   _FillValue : 1e+20
   actual_range : ( -41.18215, 38.8077 )
   add_offset : 0
   scale_factor : 1
   coordinates : lon lat level
   grid_mapping : polar_stereographic
   level_desc : 10 m
   grid_desc : polar_stereographic

I am at a loss to explain why this is happening, but I am
going to have to resolve this problem before I can
continue trying to be able to perform heavier computations
with 14 files.

Is there a way that this
script can be modified to accommodate what I am trying to
be able to do? This might be the easier route for me since
my FORTRAN abilities are not so stellar.

Any advice is appreciated. This doesn't seem like a
complicated problem compared to the other issues that are
submitted to NCL talk.

Kind regards,

DJ Rasmussen
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
Received on Sat Oct 03 2009 - 17:27:48 MDT

This archive was generated by hypermail 2.2.0 : Mon Oct 05 2009 - 13:28:34 MDT