Re: malloc error in area_hi2lores from Erik Noble on 2009-06-17 (ncl-talk 2009 archive)

From: Erik Noble <enoble_at_nyahnyahspammersnyahnyah>
Date: Wed, 17 Jun 2009 10:27:11 -0400

Hi Mary,I am now trying this out on my machine which has 18 GB of memory on
a similar set of precipitation data. But it keeps crashing (ncl 67> prate_po
= area_hi2lores_Wrap (prate&lon,prate&lat, prate, True, 1, LON, LAT,
False)
ncl(41082) malloc: *** mmap(size=1327104000) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Bus error. )

I don't have the option of working with smaller data sets.

I even tried your first and second suggestion, but it still crashed (below).

Even though I have a lot of memory, it crashes.
-Erik

noble:PERSIANN enoble$ ncl
Copyright (C) 1995-2009 - All Rights Reserved
University Corporation for Atmospheric Research
NCAR Command Language Version 5.1.0
The use of this software is governed by a License Agreement.
See http://www.ncl.ucar.edu/ for more details.
ncl 0> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
ncl 1> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
ncl 2> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
ncl 3> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/shea_util.ncl"
ncl 4> load "$NCARG_ROOT/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl"
ncl 5> ;*******************************************
ncl 6> ; PERSIANN spatial info
ncl 7> ;*******************************************
ncl 8> nlat = 480
ncl 9> mlon = 1440
ncl 10> lat = 59.875 - ispan(0,nlat-1,1)*0.25
ncl 11> lon = 0.125 + ispan(0,mlon-1,1)*0.25 ;
0.125-ispan(0,mlon-1,1)*0.25
ncl 12>
ncl 13> lat!0 = "lat"
ncl 14> lat&lat = lat
ncl 15> lat_at_units = "degrees_north"
ncl 16>
ncl 17> lon!0 = "lon"
ncl 18> lon&lon = lon
ncl 19> lon_at_units = "degrees_east"
ncl 20> ;*******************************************
ncl 21> ; Read Persiann data
ncl 22> ;*******************************************
ncl 23> setfileoption("bin","ReadByteOrder","BigEndian")
ncl 24> fils = systemfunc("ls m6s4_3h06*.bin")
ncl 25> nfils = dimsizes(fils)
ncl 26> prate = new((/nfils,nlat,mlon/),"float") ; New array for rain
rate
ncl 27> p = new((/nfils,nlat,mlon/),"float") ; New array for
rain
ncl 28> do gg = 0,dimsizes(fils)-1 ; Do loop to extract variable
form binary files
ncl 29> prate(gg,:,:) = fbindirread(fils(gg),0,(/nlat,mlon/),"float")
ncl 30> p(gg,:,:) = fbindirread(fils(gg),0,(/nlat,mlon/),"float") * 3 ; rain
per three hours (mm)
ncl 31> end do
ncl 32> prate = where(prate.lt.0.01 , 0.0, prate) ; keep precipitation
values above zero
ncl 33> p = where(p.lt.0.01 , 0.0, p)
ncl 34> ;;;;;Assign Meta Data Information;;;;;;;;;;
ncl 35>
ncl 36> prate!0 = "time"
ncl 37> prate&time = ispan(1,dimsizes(fils),1)
ncl 38> prate!1 = "lat"
ncl 39> prate&lat = lat
ncl 40> prate!2 = "lon"
ncl 41> prate&lon = lon
ncl 42> printVarSummary(prate)

Variable: prate
Type: float
Total Size: 663552000 bytes
            165888000 values
Number of Dimensions: 3
Dimensions and sizes: [time | 240] x [lat | 480] x [lon | 1440]
Coordinates:
            time: [1..240]
            lat: [59.875..-59.875]
            lon: [0.125..359.875]
Number Of Attributes: 1
  _FillValue : -999
ncl 43> prate = prate(:,::-1,:) ; Flip latitudes
ncl 44> prate = lonFlip( prate )
ncl 45> printVarSummary(prate)

Variable: prate
Type: float
Total Size: 663552000 bytes
            165888000 values
Number of Dimensions: 3
Dimensions and sizes: [time | 240] x [lat | 480] x [lon | 1440]
Coordinates:
            time: [1..240]
            lat: [-59.875..59.875]
            lon: [-179.875..179.875]
Number Of Attributes: 2
  lonFlip : longitude coordinate variable has been reordered via lonFlip
  _FillValue : -999
ncl 46> prate = lonFlip( prate )
ncl 47> prate = lonFlip( prate )
ncl 48> rm= systemfunc("rm PERSIANN_SOP3_0.25x0.25_res_3hr.nc") ;remove
the file we are about to create
rm: PERSIANN_SOP3_0.25x0.25_res_3hr.nc: No such file or directory
ncl 49> b = addfile("PERSIANN_SOP3_0.25x0.25_res_3hr.nc","c")
ncl 50> b_at_title = "1 month of daily accumulated precipitation"
ncl 51> filedimdef(b,"time",-1,True) ; make time an UNLIMITED dimension,
always recommended
ncl 52> b->persiann_rate=prate
ncl 53> b->persiann=p
ncl 54> b->lat = lat
ncl 55> b->lon = lon
ncl 57> NLAT = 240
ncl 58> MLON = 720;
ncl 59> LAT = 59.875 - ispan(0,NLAT-1,1)*0.50
ncl 60> LON = 0.125 + ispan(0,MLON-1,1)*0.50
ncl 61> LAT!0 = "lat"
ncl 62> LAT_at_units = "degrees_north"
ncl 63> LAT&lat = LAT
ncl 64> LON!0 = "lon"
ncl 65> LON_at_units = "degrees_east"
ncl 66> printVarSummary(prate)

****Lastly, 3rd suggestion******

Variable: nl
Type: integer
Total Size: 4 bytes
1 values
Number of Dimensions: 1
Dimensions and sizes: [1]
Coordinates:
(0) 240
ncl 13> do nt = 0,nl-1 ; interpolate one grid at a time
ncl 14> pi = a->persiann_rate(nt,:,:)
ncl 15> po(nt,:,:) = area_hi2lores_Wrap (pi&lon,pi&lat, pi , True, 1,
Lon,Lat,False)
ncl 16> end do
fatal:Dimension size mismatch on subscript #1, left-hand and right-hand side
dimensions do not match
fatal:Execute: Error occurred at or near line 17

ncl 19>
????

On Fri, Jun 12, 2009 at 11:54 AM, Mary Haley <haley_at_ucar.edu> wrote:

> Erik,
>
> I think you are simply running out of memory on your machine. How
> much memory do you have on your machine?
>
> What happens if you only execute these lines:
>
> a = addfile("CMORPH_September_.25x.25.nc","r")
> p = a->CMOPRH_RAIN_RATE
> p =p(:,::-1,:)
> Lat = fspan(-59.875,59.875,240)
> Lon = fspan(-180,180,720)
> po = area_hi2lores (p&lon,p&lat, p , True, 1, Lon, Lat, False)
>
> If the above fails, what about this:
>
> a = addfile("CMORPH_September_.25x.25.nc","r")
> p = a->CMOPRH_RAIN_RATE
> p =p(:,::-1,:)
> nl = dimsizes(p(:,0,0))
>
> Lat = fspan(-59.875,59.875,240)
> Lon = fspan(-180,180,720)
> po = new((/nl,720,240/),typeof(p))
>
> --Mary
>
> On Fri, 12 Jun 2009, Erik Noble wrote:
>
> > Hi. I fixed those errors. Lat variable is monotonically increasing (ncl
> 7>
> > rain_new=p(:,::-1,:) ) and the lat and lon variables are not transposed
> in
> > the are_hi2low function.Thank you for catching those two.
> >
> > But I am still getting this error.
> >
> >
> > noble:CMORPH enoble$ ncl
> > Copyright (C) 1995-2009 - All Rights Reserved
> > University Corporation for Atmospheric Research
> > NCAR Command Language Version 5.1.0
> > The use of this software is governed by a License Agreement.
> > See http://www.ncl.ucar.edu/ for more details.
> > ncl 0> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
> > ncl 1> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
> > ncl 2> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
> > ncl 3> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/shea_util.ncl"
> > ncl 4> load "$NCARG_ROOT/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl"
> > ncl 5> a = addfile("CMORPH_September_.25x.25.nc","r")
> > ncl 6> p = a->CMOPRH_RAIN_RATE
> > ncl 7> p =p(:,::-1,:)
> > ncl 8> Lat = fspan(-59.875,59.875,240)
> > ncl 9> Lon = fspan(-180,180,720)
> > ncl 10> po = area_hi2lores_Wrap (p&lon,p&lat, p , True, 1, Lon, Lat,
> > False)
> > ncl(96407) malloc: *** mmap(size=1327104000) failed (error code=12)
> > *** error: can't allocate region
> > *** set a breakpoint in malloc_error_break to debug
> > Bus error
> > *****************************************
> >
> > BELOW,I used the example from the NCL website and it worked (below). But
> > when I try to interpolate the to the grid I need, it gives me an error
> > (above.)
> >
> > noble:CMORPH enoble$ ncl
> > Copyright (C) 1995-2009 - All Rights Reserved
> > University Corporation for Atmospheric Research
> > NCAR Command Language Version 5.1.0
> > The use of this software is governed by a License Agreement.
> > See http://www.ncl.ucar.edu/ for more details.
> > ncl 0> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
> > ncl 1> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
> > ncl 2> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
> > ncl 3> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/shea_util.ncl"
> > ncl 4> load "$NCARG_ROOT/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl"
> > ncl 5> a = addfile("CMORPH_September_.25x.25.nc","r")
> > ncl 6> p = a->CMOPRH_RAIN_RATE
> > ncl 7> p =p(:,::-1,:) ; make lat monotonically increasing
> > ncl 8> LON = fspan(-178.5,178.5, 120)
> > ncl 9> LAT = fspan(-48,48, 49)
> > ncl 10> po = area_hi2lores_Wrap (p&lon,p&lat, p , True, 1, LON, LAT,
> > False) ; (ntim,49,120)
> > ncl 11> printMinMax(po, True)
> > (0)
> > (0) CMORPH: min=0 max=11.5147
> > ncl 12> printMinMax(p, True)
> > (0)
> > (0) CMORPH: min=0 max=40.3708
> >
> >
> >>
> >> _______________________________________________
> >> ncl-talk mailing list
> >> List instructions, subscriber options, unsubscribe:
> >> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
> >>
> >
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
Received on Wed Jun 17 2009 - 08:27:11 MDT

This archive was generated by hypermail 2.2.0 : Fri Jun 19 2009 - 13:23:25 MDT