Re: NCL can't perform one simple average with large file? Memory Issue

DJ Rasmussen,

Can you check your machine's total memory first?

Here, your variable UAS takes about 1G.
And you need to check if there is other processes
tool lot of memory as well.

Thanks,

Wei Huang
huangwei_at_ucar.edu
VETS/CISL
National Center for Atmospheric Research
P.O. Box 3000 (1850 Table Mesa Dr.)
Boulder, CO 80307-3000 USA
(303) 497-8924

On Oct 3, 2009, at 5:27 PM, DJ Rasmussen wrote:

> NCL Community,
>
> I have 14 u and v wind component CRCM files that I would
> like to average. I have been trying to create a script
> that reads the files in and performs one giant average of
> all of the data, but I am running into memory errors. Even
> averaging just one file alone gives an error with this
> script:
>
> Error: fatal:NclMalloc Failed:[errno=12]
>
> Script:
> ;*************************************************
> ;NARCCAPAVG.ncl
> ;
> ;************************************************
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
> load
> "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
>
> ;************************************************
> begin
>
> ;************************************************
> ; read in multiple netCDF files
> ;************************************************
> diri = "/home/rasmussen/NARCCAP/netCDF/RCM3/" ; directory
> where files reside
> fils = systemfunc("ls "+diri+"uas_CRCM_*.nc")
> nfil = dimsizes(fils)
> print(fils)
>
> ;*************************Get Variables**********
> f = addfiles (fils, "r")
>
> UAS = f[:]->uas ; (time,yc,xc)
>
> printVarSummary(UAS)
>
> ;*************************Calculate Average******
>
> WSPD_AVG = dim_avg_n(UAS,0) ; (lat,lon)
> WSPD_AVG_at_long_name = "mean wind speed"
>
> printVarSummary(WSPD_AVG)
>
> ;************************************************
> ; output ascii
> ;************************************************
>
> fName = "RCM3AVG.txt"
> system("/bin/rm -f "+fName) ; remove if file is present
> asciiwrite (fName , WSPD_AVG)
>
> end
>
> The basic script can read smaller files with less time
> steps just fine (8760 steps), but it gives an error with
> the files that have 14600 time steps. (3hrly over 5 years)
>
> Variable: UAS
> Type: float
> Total Size: 940240000 bytes
> 235060000 values
> Number of Dimensions: 3
> Dimensions and sizes: [time | 14600] x [yc | 115] x [xc |
> 140]
> Coordinates:
> time: [8395.125..10220]
> yc: [ 0..5700000]
> xc: [ 0..6950000]
> Number Of Attributes: 13
> units : m s-1
> type : instantaneaous
> long_name : Zonal Surface Wind Speed
> standard_name : eastward_wind
> missing_value : 1e+20
> _FillValue : 1e+20
> actual_range : ( -41.18215, 38.8077 )
> add_offset : 0
> scale_factor : 1
> coordinates : lon lat level
> grid_mapping : polar_stereographic
> level_desc : 10 m
> grid_desc : polar_stereographic
>
> I am at a loss to explain why this is happening, but I am
> going to have to resolve this problem before I can
> continue trying to be able to perform heavier computations
> with 14 files.
>
> Is there a way that this
> http://www.ncl.ucar.edu/Applications/Scripts/crcm_1.ncl
> script can be modified to accommodate what I am trying to
> be able to do? This might be the easier route for me since
> my FORTRAN abilities are not so stellar.
>
> Any advice is appreciated. This doesn't seem like a
> complicated problem compared to the other issues that are
> submitted to NCL talk.
>
> Kind regards,
>
> DJ Rasmussen
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Mon Oct 05 2009 - 08:20:54 MDT