Erik,
We are going to take this offline after this message.
Just because you have 18 Gb of memory, doesn't mean you can access
this all at once. If you are on a 32-bit system, you are going to
have more limitations on how much memory you can grab than if
you are on a 64-bit system.
Dennis has already asked for your file offline, and we'll take it
from there.
--Mary
On Wed, 17 Jun 2009, Erik Noble wrote:
> Hi Mary,I am now trying this out on my machine which has 18 GB of memory on
> a similar set of precipitation data. But it keeps crashing (ncl 67> prate_po
> = area_hi2lores_Wrap (prate&lon,prate&lat, prate, True, 1, LON, LAT,
> False)
> ncl(41082) malloc: *** mmap(size=1327104000) failed (error code=12)
> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug
> Bus error. )
>
> I don't have the option of working with smaller data sets.
>
> I even tried your first and second suggestion, but it still crashed (below).
>
> Even though I have a lot of memory, it crashes.
> -Erik
>
> noble:PERSIANN enoble$ ncl
> Copyright (C) 1995-2009 - All Rights Reserved
> University Corporation for Atmospheric Research
> NCAR Command Language Version 5.1.0
> The use of this software is governed by a License Agreement.
> See http://www.ncl.ucar.edu/ for more details.
> ncl 0> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
> ncl 1> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
> ncl 2> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
> ncl 3> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/shea_util.ncl"
> ncl 4> load "$NCARG_ROOT/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl"
> ncl 5> ;*******************************************
> ncl 6> ; PERSIANN spatial info
> ncl 7> ;*******************************************
> ncl 8> nlat = 480
> ncl 9> mlon = 1440
> ncl 10> lat = 59.875 - ispan(0,nlat-1,1)*0.25
> ncl 11> lon = 0.125 + ispan(0,mlon-1,1)*0.25 ;
> 0.125-ispan(0,mlon-1,1)*0.25
> ncl 12>
> ncl 13> lat!0 = "lat"
> ncl 14> lat&lat = lat
> ncl 15> lat_at_units = "degrees_north"
> ncl 16>
> ncl 17> lon!0 = "lon"
> ncl 18> lon&lon = lon
> ncl 19> lon_at_units = "degrees_east"
> ncl 20> ;*******************************************
> ncl 21> ; Read Persiann data
> ncl 22> ;*******************************************
> ncl 23> setfileoption("bin","ReadByteOrder","BigEndian")
> ncl 24> fils = systemfunc("ls m6s4_3h06*.bin")
> ncl 25> nfils = dimsizes(fils)
> ncl 26> prate = new((/nfils,nlat,mlon/),"float") ; New array for rain
> rate
> ncl 27> p = new((/nfils,nlat,mlon/),"float") ; New array for
> rain
> ncl 28> do gg = 0,dimsizes(fils)-1 ; Do loop to extract variable
> form binary files
> ncl 29> prate(gg,:,:) = fbindirread(fils(gg),0,(/nlat,mlon/),"float")
> ncl 30> p(gg,:,:) = fbindirread(fils(gg),0,(/nlat,mlon/),"float") * 3 ; rain
> per three hours (mm)
> ncl 31> end do
> ncl 32> prate = where(prate.lt.0.01 , 0.0, prate) ; keep precipitation
> values above zero
> ncl 33> p = where(p.lt.0.01 , 0.0, p)
> ncl 34> ;;;;;Assign Meta Data Information;;;;;;;;;;
> ncl 35>
> ncl 36> prate!0 = "time"
> ncl 37> prate&time = ispan(1,dimsizes(fils),1)
> ncl 38> prate!1 = "lat"
> ncl 39> prate&lat = lat
> ncl 40> prate!2 = "lon"
> ncl 41> prate&lon = lon
> ncl 42> printVarSummary(prate)
>
>
> Variable: prate
> Type: float
> Total Size: 663552000 bytes
> 165888000 values
> Number of Dimensions: 3
> Dimensions and sizes: [time | 240] x [lat | 480] x [lon | 1440]
> Coordinates:
> time: [1..240]
> lat: [59.875..-59.875]
> lon: [0.125..359.875]
> Number Of Attributes: 1
> _FillValue : -999
> ncl 43> prate = prate(:,::-1,:) ; Flip latitudes
> ncl 44> prate = lonFlip( prate )
> ncl 45> printVarSummary(prate)
>
>
> Variable: prate
> Type: float
> Total Size: 663552000 bytes
> 165888000 values
> Number of Dimensions: 3
> Dimensions and sizes: [time | 240] x [lat | 480] x [lon | 1440]
> Coordinates:
> time: [1..240]
> lat: [-59.875..59.875]
> lon: [-179.875..179.875]
> Number Of Attributes: 2
> lonFlip : longitude coordinate variable has been reordered via lonFlip
> _FillValue : -999
> ncl 46> prate = lonFlip( prate )
> ncl 47> prate = lonFlip( prate )
> ncl 48> rm= systemfunc("rm PERSIANN_SOP3_0.25x0.25_res_3hr.nc") ;remove
> the file we are about to create
> rm: PERSIANN_SOP3_0.25x0.25_res_3hr.nc: No such file or directory
> ncl 49> b = addfile("PERSIANN_SOP3_0.25x0.25_res_3hr.nc","c")
> ncl 50> b_at_title = "1 month of daily accumulated precipitation"
> ncl 51> filedimdef(b,"time",-1,True) ; make time an UNLIMITED dimension,
> always recommended
> ncl 52> b->persiann_rate=prate
> ncl 53> b->persiann=p
> ncl 54> b->lat = lat
> ncl 55> b->lon = lon
> ncl 57> NLAT = 240
> ncl 58> MLON = 720;
> ncl 59> LAT = 59.875 - ispan(0,NLAT-1,1)*0.50
> ncl 60> LON = 0.125 + ispan(0,MLON-1,1)*0.50
> ncl 61> LAT!0 = "lat"
> ncl 62> LAT_at_units = "degrees_north"
> ncl 63> LAT&lat = LAT
> ncl 64> LON!0 = "lon"
> ncl 65> LON_at_units = "degrees_east"
> ncl 66> printVarSummary(prate)
>
>
> Variable: prate
> Type: float
> Total Size: 663552000 bytes
> 165888000 values
> Number of Dimensions: 3
> Dimensions and sizes: [time | 240] x [lat | 480] x [lon | 1440]
> Coordinates:
> time: [1..240]
> lat: [-59.875..59.875]
> lon: [-179.875..179.875]
> Number Of Attributes: 2
> lonFlip : longitude coordinate variable has been reordered via lonFlip
> _FillValue : -999
> ncl 67> prate_po = area_hi2lores_Wrap (prate&lon,prate&lat, prate, True, 1,
> LON, LAT, False)
> ncl(41082) malloc: *** mmap(size=1327104000) failed (error code=12)
> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug
> Bus error
> noble:PERSIANN enoble$
> *************************************************************
> ************************************************************
> ************************************************************
>
> noble:PERSIANN enoble$ ncl
> Copyright (C) 1995-2009 - All Rights Reserved
> University Corporation for Atmospheric Research
> NCAR Command Language Version 5.1.0
> The use of this software is governed by a License Agreement.
> See http://www.ncl.ucar.edu/ for more details.
> ncl 0> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
> ncl 1> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
> ncl 2> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
> ncl 3> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/shea_util.ncl"
> ncl 4> load "$NCARG_ROOT/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl"
> ncl 5> a = addfile("PERSIANN_SOP3_0.25x0.25_res_3hr.nc","r")
> ncl 6> p = a->persiann_rate
> ncl 7> p =p(:,::-1,:)
> ncl 8> Lon = 0.125 + ispan(0,719,1)*0.50
> ncl 9> Lat = 59.875 + ispan(0,239,1)*0.50
> ncl 10> po = area_hi2lores (p&lon,p&lat, p , True, 1, Lon, Lat, False)
> ncl(44059) malloc: *** mmap(size=1327104000) failed (error code=12)
> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug
> Bus error
>
>
> ****Lastly, 3rd suggestion******
>
> noble:PERSIANN enoble$ ncl
> Copyright (C) 1995-2009 - All Rights Reserved
> University Corporation for Atmospheric Research
> NCAR Command Language Version 5.1.0
> The use of this software is governed by a License Agreement.
> See http://www.ncl.ucar.edu/ for more details.
> ncl 0> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
> ncl 1> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
> ncl 2> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
> ncl 3> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/shea_util.ncl"
> ncl 4> load "$NCARG_ROOT/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl"
> ncl 5> a = addfile("PERSIANN_SOP3_0.25x0.25_res_3hr.nc","r")
> ncl 6> p = a->persiann_rate
> ncl 7> p =p(:,::-1,:)
> ncl 8> nl = dimsizes(p(:,0,0))
> ncl 9> Lat = 59.875 + ispan(0,239,1)*0.50
> ncl 10> Lon = 0.125 + ispan(0,719,1)*0.50
> ncl 11> po = new((/nl,720,240/),typeof(p))
> ncl 12> print(nl)
>
>
> Variable: nl
> Type: integer
> Total Size: 4 bytes
> 1 values
> Number of Dimensions: 1
> Dimensions and sizes: [1]
> Coordinates:
> (0) 240
> ncl 13> do nt = 0,nl-1 ; interpolate one grid at a time
> ncl 14> pi = a->persiann_rate(nt,:,:)
> ncl 15> po(nt,:,:) = area_hi2lores_Wrap (pi&lon,pi&lat, pi , True, 1,
> Lon,Lat,False)
> ncl 16> end do
> fatal:Dimension size mismatch on subscript #1, left-hand and right-hand side
> dimensions do not match
> fatal:Execute: Error occurred at or near line 17
>
> ncl 19>
> ????
>
>
> On Fri, Jun 12, 2009 at 11:54 AM, Mary Haley <haley_at_ucar.edu> wrote:
>
>> Erik,
>>
>> I think you are simply running out of memory on your machine. How
>> much memory do you have on your machine?
>>
>> What happens if you only execute these lines:
>>
>> a = addfile("CMORPH_September_.25x.25.nc","r")
>> p = a->CMOPRH_RAIN_RATE
>> p =p(:,::-1,:)
>> Lat = fspan(-59.875,59.875,240)
>> Lon = fspan(-180,180,720)
>> po = area_hi2lores (p&lon,p&lat, p , True, 1, Lon, Lat, False)
>>
>> If the above fails, what about this:
>>
>> a = addfile("CMORPH_September_.25x.25.nc","r")
>> p = a->CMOPRH_RAIN_RATE
>> p =p(:,::-1,:)
>> nl = dimsizes(p(:,0,0))
>>
>> Lat = fspan(-59.875,59.875,240)
>> Lon = fspan(-180,180,720)
>> po = new((/nl,720,240/),typeof(p))
>>
>> --Mary
>>
>> On Fri, 12 Jun 2009, Erik Noble wrote:
>>
>>> Hi. I fixed those errors. Lat variable is monotonically increasing (ncl
>> 7>
>>> rain_new=p(:,::-1,:) ) and the lat and lon variables are not transposed
>> in
>>> the are_hi2low function.Thank you for catching those two.
>>>
>>> But I am still getting this error.
>>>
>>>
>>> noble:CMORPH enoble$ ncl
>>> Copyright (C) 1995-2009 - All Rights Reserved
>>> University Corporation for Atmospheric Research
>>> NCAR Command Language Version 5.1.0
>>> The use of this software is governed by a License Agreement.
>>> See http://www.ncl.ucar.edu/ for more details.
>>> ncl 0> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
>>> ncl 1> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
>>> ncl 2> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
>>> ncl 3> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/shea_util.ncl"
>>> ncl 4> load "$NCARG_ROOT/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl"
>>> ncl 5> a = addfile("CMORPH_September_.25x.25.nc","r")
>>> ncl 6> p = a->CMOPRH_RAIN_RATE
>>> ncl 7> p =p(:,::-1,:)
>>> ncl 8> Lat = fspan(-59.875,59.875,240)
>>> ncl 9> Lon = fspan(-180,180,720)
>>> ncl 10> po = area_hi2lores_Wrap (p&lon,p&lat, p , True, 1, Lon, Lat,
>>> False)
>>> ncl(96407) malloc: *** mmap(size=1327104000) failed (error code=12)
>>> *** error: can't allocate region
>>> *** set a breakpoint in malloc_error_break to debug
>>> Bus error
>>> *****************************************
>>>
>>> BELOW,I used the example from the NCL website and it worked (below). But
>>> when I try to interpolate the to the grid I need, it gives me an error
>>> (above.)
>>>
>>> noble:CMORPH enoble$ ncl
>>> Copyright (C) 1995-2009 - All Rights Reserved
>>> University Corporation for Atmospheric Research
>>> NCAR Command Language Version 5.1.0
>>> The use of this software is governed by a License Agreement.
>>> See http://www.ncl.ucar.edu/ for more details.
>>> ncl 0> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
>>> ncl 1> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
>>> ncl 2> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
>>> ncl 3> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/shea_util.ncl"
>>> ncl 4> load "$NCARG_ROOT/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl"
>>> ncl 5> a = addfile("CMORPH_September_.25x.25.nc","r")
>>> ncl 6> p = a->CMOPRH_RAIN_RATE
>>> ncl 7> p =p(:,::-1,:) ; make lat monotonically increasing
>>> ncl 8> LON = fspan(-178.5,178.5, 120)
>>> ncl 9> LAT = fspan(-48,48, 49)
>>> ncl 10> po = area_hi2lores_Wrap (p&lon,p&lat, p , True, 1, LON, LAT,
>>> False) ; (ntim,49,120)
>>> ncl 11> printMinMax(po, True)
>>> (0)
>>> (0) CMORPH: min=0 max=11.5147
>>> ncl 12> printMinMax(p, True)
>>> (0)
>>> (0) CMORPH: min=0 max=40.3708
>>>
>>>
>>>>
>>>> _______________________________________________
>>>> ncl-talk mailing list
>>>> List instructions, subscriber options, unsubscribe:
>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>
>>>
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>
_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Thu Jun 18 2009 - 07:02:30 MDT
This archive was generated by hypermail 2.2.0 : Fri Jun 19 2009 - 13:23:25 MDT