Re: NclMalloc Failed:[errno=12]...switch to float?

From: Dave Allured <dave.allured_at_nyahnyahspammersnyahnyah>
Date: Tue, 22 Sep 2009 17:54:25 -0600

DJ,

Try this. Start with the systemfunc function to get your list of
input files as a string array. Loop over all input files, and read
them one at a time. Use addfile for this, not addfiles.

Initialize two lat/lon arrays to zero before the loop. One is the
cumulative sum, the other is the cumulative data value count. The
count array is not needed if you have no missing values anywhere.

For each file, compute wind speed magnitude for just the time steps
in that file. Then use dim_sum_?? to get lat/lon sum of magnitude
and data count for just that file. Then add the file sum and count
grids to the cumulative grids. End of loop.

Be sure to delete (f) at the end of each loop to close the current
input file before starting the next one. You might also need to
delete intermediate arrays between files if they change dimensions.

After the file loop, simply compute final averaged wind from the two
cumulative arrays: sum(x,y) divided by count(x,y).

--Dave

DJ Rasmussen wrote:
> Dennis, Dave, and NCL users,
>
> It seems deleting the unused arrays UAS and VAS is still not enough to
> free up memory on my system. I also tried just averaging UAS, the u
> component wind, and NCL gave the same error.
>
> If I were to implement a do loop, as Dave has suggested, I am not sure
> where to even add it since I cannot even average one wind component. Is
> there anywhere else in my code where I could free up some additional memory?
>
> I appreciate any other assistance in this particular problem.
>
> DJ
>
> Dennis Shea wrote:
>> Dave is correct.
>>
>> Mitigation:
>>
>> WSPD = sqrt(UAS^2 + VAS^2) ; calculate wind speed magnitude
>> copy_VarMeta(UAS, WSPD)
>> delete(UAS) ; free memory
>> delete(VAS)
>>
>>
>> FYI:
>> NCL does *all* computations in double precision.
>> The underlying interface to the computational code promotes the float
>> input array
>> to type double.
>>
>>
>> Dave Allured wrote:
>>> DJ,
>>>
>>> This looks like just a normal memory limit problem. This probably
>>> has nothing to do with the data type. The WPSD array is huge, 1/2
>>> Gb. And that is on top of other large arrays that you had already
>>> allocated. You probably exceeded memory available to NCL on the
>>> first statement that tried to create WPSD.
>>>
>>> Best practice for this kind of problem is to divide the computation
>>> into increments or slices along one dimension, whatever would be the
>>> simplest extension of your existing code. This is an example of
>>> where adding one do loop is appropriate.
>>>
>>> You could also try the unix unlimit command to increase available
>>> memory, but that may not work when you are approaching physical
>>> memory limits. HTH.
>>>
>>> Dave Allured
>>> CU/CIRES Climate Diagnostics Center (CDC)
>>> http://cires.colorado.edu/science/centers/cdc/
>>> NOAA/ESRL/PSD, Climate Analysis Branch (CAB)
>>> http://www.cdc.noaa.gov/psd1/
>>>
>>> DJ Rasmussen wrote:
>>>
>>>> I have a NCL file that calls in some files (up to 14) then computes
>>>> some averages of them. However, with NCL version 5.1.1 I get the
>>>> following error:
>>>>
>>>> fatal:NclMalloc Failed:[errno=12]
>>>> fatal:dim_avg_n: Could not coerce input data to double, can't continue
>>>> fatal:Execute: Error occurred at or near line 40 in file NARCCAPAVG.ncl
>>>>
>>>> Which I know has something to do with memory allocation. I am
>>>> guessing I need to have my variables somehow be set to float when I
>>>> use the function dim_avg_n_Wrap. Here is a print out of my variable,
>>>> WSPD:
>>>>
>>>> Variable: WSPD
>>>> Type: float
>>>> Total Size: 488317440 bytes
>>>> 122079360 values
>>>> Number of Dimensions: 3
>>>> Dimensions and sizes: [time | 8760] x [yc | 104] x [xc | 134]
>>>> Coordinates:
>>>> time: [0.125..1096]
>>>> yc: [600000..5750000]
>>>> xc: [600000..7250000]
>>>> Number Of Attributes: 9
>>>> _FillValue : 1e+20
>>>> original_units : m s-1
>>>> original_name : UA
>>>> missing_value : 1e+20
>>>> coordinates : lon lat level
>>>> units : m s-1
>>>> long_name : wind speed
>>>> standard_name : eastward_wind
>>>> grid_mapping : Transverse_Mercator
>>>>
>>>>
>>>> Looking at the NCL archive, I didn't see anything related to this
>>>> particular problem with this function, dim_avg_n_Wrap. Maybe because
>>>> it is rather new?
>>>>
>>>> Any advice would be helpful.
>>>>
>>>>
>>>> My code:
>>>>
>>>> ;*************************************************
>>>> ;NARCCAPAVG.ncl
>>>> ;
>>>> ;************************************************
>>>> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
>>>> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
>>>> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
>>>>
>>>> ;************************************************
>>>> begin
>>>>
>>>> ;************************************************
>>>> ; read in multiple netCDF files
>>>> ;************************************************
>>>> diri = "/home/rasmussen/NARCCAP/netCDF/RCM3/" ; directory where
>>>> files reside
>>>> fils = systemfunc("ls "+diri+"*as_RCM3_*.nc")
>>>> nfil = dimsizes(fils)
>>>> print(fils)
>>>>
>>>> ;*************************Get Variables**********
>>>> f = addfiles (fils, "r")
>>>>
>>>> UAS = f[:]->uas ; (time,yc,xc)
>>>> VAS = f[:]->vas ; (time,yc,xc)
>>>>
>>>> printVarSummary(UAS)
>>>> printVarSummary(VAS)
>>>>
>>>> WSPD = sqrt(UAS^2 + VAS^2) ; calculate wind speed magnitude
>>>>
>>>> WSPD=(WSPD*2.2) ; convert to miles per hour
>>>>
>>>> copy_VarMeta(UAS, WSPD) ; copy dimensions to new averaged wind
>>>> speed array
>>>> WSPD_at_long_name = "wind speed"
>>>> printVarSummary(WSPD)
>>>>
>>>> ;*************************Calculate Average******
>>>>
>>>> WSPD_AVG = dim_avg_n_Wrap(WSPD,0) ; (lat,lon)
>>>> WSPD_AVG_at_long_name = "mean wind speed"
>>>>
>>>> printVarSummary(WSPD_AVG)
>>>>
>>>> ;************************************************
>>>> ; output ascii
>>>> ;************************************************
>>>>
>>>> fName = "RCM3AVG.txt"
>>>> system("/bin/rm -f "+fName) ; remove if file is present
>>>> asciiwrite (fName , WSPD)
>>>>
>>>> end
>>>> _______________________________________________
>>>> ncl-talk mailing list
>>>> List instructions, subscriber options, unsubscribe:
>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>
>>> _______________________________________________
>>> ncl-talk mailing list
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>
>
_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Tue Sep 22 2009 - 17:54:25 MDT

This archive was generated by hypermail 2.2.0 : Tue Sep 29 2009 - 10:11:39 MDT