Re: NCL can't perform one simple average with large file? Memory Issue

From: Adam Phillips <asphilli_at_nyahnyahspammersnyahnyah>
Date: Mon, 05 Oct 2009 11:14:44 -0600

Hi DJ,
When you get a NclMalloc error it generally
means you are running out of memory, as Wei
alluded to. You might try using the netCDF
operator ncra:
http://nco.sourceforge.net/nco.html#ncra-netCDF-Record-Averager

This allows you to create an average outside of
the NCL framework, although I seriously doubt
that NCL is the cause of your memory issues. You
will likely have to split your data up, as your
array size sounds like it is too big for the
machine you are working on. (Also note that NCL
can not handle an array > 2GB.)
Best regards,
Adam

DJ Rasmussen wrote:
> NCL Community,
>
> I have 14 u and v wind component CRCM files that I would
> like to average. I have been trying to create a script
> that reads the files in and performs one giant average of
> all of the data, but I am running into memory errors. Even
> averaging just one file alone gives an error with this
> script:
>
> Error: fatal:NclMalloc Failed:[errno=12]
>
> Script:
> ;*************************************************
> ;NARCCAPAVG.ncl
> ;
> ;************************************************
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
> load
> "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
>
> ;************************************************
> begin
>
> ;************************************************
> ; read in multiple netCDF files
> ;************************************************
> diri = "/home/rasmussen/NARCCAP/netCDF/RCM3/" ; directory
> where files reside
> fils = systemfunc("ls "+diri+"uas_CRCM_*.nc")
> nfil = dimsizes(fils)
> print(fils)
>
> ;*************************Get Variables**********
> f = addfiles (fils, "r")
>
> UAS = f[:]->uas ; (time,yc,xc)
>
> printVarSummary(UAS)
>
> ;*************************Calculate Average******
>
> WSPD_AVG = dim_avg_n(UAS,0) ; (lat,lon)
> WSPD_AVG_at_long_name = "mean wind speed"
>
> printVarSummary(WSPD_AVG)
>
> ;************************************************
> ; output ascii
> ;************************************************
>
> fName = "RCM3AVG.txt"
> system("/bin/rm -f "+fName) ; remove if file is present
> asciiwrite (fName , WSPD_AVG)
>
> end
>
> The basic script can read smaller files with less time
> steps just fine (8760 steps), but it gives an error with
> the files that have 14600 time steps. (3hrly over 5 years)
>
> Variable: UAS
> Type: float
> Total Size: 940240000 bytes
> 235060000 values
> Number of Dimensions: 3
> Dimensions and sizes: [time | 14600] x [yc | 115] x [xc |
> 140]
> Coordinates:
> time: [8395.125..10220]
> yc: [ 0..5700000]
> xc: [ 0..6950000]
> Number Of Attributes: 13
> units : m s-1
> type : instantaneaous
> long_name : Zonal Surface Wind Speed
> standard_name : eastward_wind
> missing_value : 1e+20
> _FillValue : 1e+20
> actual_range : ( -41.18215, 38.8077 )
> add_offset : 0
> scale_factor : 1
> coordinates : lon lat level
> grid_mapping : polar_stereographic
> level_desc : 10 m
> grid_desc : polar_stereographic
>
> I am at a loss to explain why this is happening, but I am
> going to have to resolve this problem before I can
> continue trying to be able to perform heavier computations
> with 14 files.
>
> Is there a way that this
> http://www.ncl.ucar.edu/Applications/Scripts/crcm_1.ncl
> script can be modified to accommodate what I am trying to
> be able to do? This might be the easier route for me since
> my FORTRAN abilities are not so stellar.
>
> Any advice is appreciated. This doesn't seem like a
> complicated problem compared to the other issues that are
> submitted to NCL talk.
>
> Kind regards,
>
> DJ Rasmussen
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

-- 
__________________________________________________
Adam Phillips 
                 asphilli_at_ucar.edu
National Center for Atmospheric Research   tel: 
(303) 497-1726
Climate and Global Dynamics Division 
fax: (303) 497-1333
P.O. Box 3000				
Boulder, CO 80307-3000 
http://www.cgd.ucar.edu/cas/asphilli
_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Mon Oct 05 2009 - 11:14:44 MDT

This archive was generated by hypermail 2.2.0 : Mon Oct 05 2009 - 13:28:34 MDT