Hello
I will let one of the developers answer the memory question.
----
The reason the successfully created netCDF file is twice
the size of the GRIB file is that the ECMWF GRIB is basically
a 16-bit pack with a scale and offset. When unpacked to type
float [type real in fortran], the numbers have been converted
to float which is 32-bit.
If you want a netCDF that is a 16-bit pack you could:
[1] NCL's "pack_values
http://www.ncl.ucar.edu/Document/Functions/Contributed/pack_values.shtml
[2] Take the netCDF file created by NCL and the use the
netCDF Operator (ncpdq) to pack the data.
http://nco.sourceforge.net/nco.html#ncpdq
ncpdq -P all_new -M flt_sht in.nc out.nc
[3] FYI: if using NCL to process, you do noot need to
create a netCDF file. Read the grib directly and
process.
Good luck
Yolande Serra wrote:
>
> Hi,
>
> I'm a new user (as of today) to NCL and I am trying to convert some
> GRIB1 files (1GB in size) to netCDF files to use with my own analysis
> code. I have a PowerMac G5 dual core 64-bit machine running OS X
> 10.4.11 and the latest version of NCL/NCAR graphics (installed today).
> I've used the netcdf library for several years now and have no trouble
> creating large (>2GB) netcdf files using FORTRAN code and the netcdf
> library routines. However, when I try to convert my GRIB files I'm
> getting this error:
>
> warning:FileVarAttDef: Variable (time) has no attributes to assign
> ncl(22003) malloc: *** vm_allocate(size=2279014400) failed (error code=3)
> ncl(22003) malloc: *** error: can't allocate region
> ncl(22003) malloc: *** set a breakpoint in szone_error to debug
> fatal:NclMalloc Failed:[errno=12]
> Bus error
> [cassava:ECMWF/4x/NCAR] serra%
>
> The script I am running is a makeover of one of the scripts I found on
> the NCL site for converting grib to netcdf including changing names of
> variables and dimensions which is ultimately what I'd like to do.
> Attached at the end of this message is my version of this script.
>
> I get basically the same error if I try to run ncl_convert2nc even
> with the -L option. However, if I use another script I found on your
> website it works but the resulting netCDF file is 2GB (2279031580
> bytes), twice the size of the GRIB file. Here is that script:
>
> begin
> ;***********************************************
> ; get variable names from grib file
> ;***********************************************
> grib_in = addfile("./e4oper.an.pl.t106.q.1980.grb","r")
> names = getfilevarnames(grib_in); extract all variable names
> ;***********************************************
> ; create output netcdf file
> ;***********************************************
> system("rm e4oper.an.pl.t106.q.1980.nc") ; remove any pre-existing
> file
> ncdf_out = addfile("e4oper.an.pl.t106.q.1980.nc" ,"c") ;
> create output netCDF file
> ;***********************************************
> ; loop through variables and output each to netcdf
> ;***********************************************
> do i = 1, dimsizes(names)-1
> ncdf_out->$names(i)$ = grib_in->$names(i)$
> end do
> end
>
> This script will not work if the do loop goes from 0 to
> dimsizes(names)-1 as the first variable seems to not be one that can
> be written to a netcdf file?? What is strange is that the script below
> that does not work (gives the bus error) should write a smaller netcdf
> file since I don't put out all the time variables and I change time to
> an integer from a double.
>
> Any ideas??
>
> Best regards, Yolande Serra
>
> ----------------------------------------------------------
> Yolande Serra, Research Faculty
> Department of Atmospheric Science
> 524 PAS Bldg
> 1118 E 4th St, P.O. Box 210081
> University of Arizona
> Tucson, AZ 85721-0081
> Tel: (520) 621-6619; Fax: (520) 621-6833
>
> **** begin script to convert grib to netcdf *****
>
> load "/usr/local/lib/ncarg/nclscripts/csm/contributed.ncl"
>
> function rdVar4D (f:file, vName:string, time:integer)
> ; read GRIB 4D variable and assign new dimension names
> begin
> var = f->$vName$; read variable
> var!0 = "time" ; assign 'conventional' dimension names
> var!1 = "level"
> var!2 = "lat"
> var!3 = "lon"
> var&time = time ; assign time coord variable
> return(var)
> end
>
> begin
> ;***********************************************
> ; read in data from GRIB file
> ;***********************************************
> gribdir = "./"
> gribfil = "e4oper.an.pl.t106.q.1980.grb"
> grib_in = addfile(gribdir+gribfil, "r")
>
> print( grib_in ) ; file overview
>
> TIME = grib_in->initial_time0_hours ; time as "hours since"
> lat = grib_in->g4_lat_2 ; latitude
> lon = grib_in->g4_lon_3 ; longitude
> level = grib_in->lv_ISBL1 ; level
>
> lat!0 = "lat" ; assign new dimension name
> lon!0 = "lon"
> level!0 = "level"
> ; TIME!0= "time"
>
> date = ut_calendar(TIME, 0) ; change to date syntax
> dtime =
> ut_inv_calendar(floattointeger(date(:,0)),floattointeger(date(:,1)),floattointeger(date(:,2)),floattointeger(date(:,3)),floattointeger(date(:,4)),date(:,5),"hours
> since 1-1-1 00:00:0.0",0)
> time = doubletointeger(dtime)
> time!0 = "time"
>
> shum = rdVar4D (grib_in, "Q_GDS4_ISBL" ,time)
>
> ;***********************************************
> ; create parameters and output file
> ;***********************************************
> ntim = dimsizes(time)
> klev = dimsizes(level)
> nlat = dimsizes(lat)
> mlon = dimsizes(lon)
>
> system("'rm' -f e4oper.an.pl.t106.q.1980.nc") ; remove
> pre-exit file (if any)
> ncdf = addfile("e4oper.an.pl.t106.q.1980.nc","c") ; create
> output file
>
> ;***************************************************
> ; define file options [Version a033]
> ;***********************************************
> setfileoption(ncdf, "prefill", False)
> setfileoption(ncdf, "suppressclose", True)
> setfileoption(ncdf, "definemode", True)
>
> ;***********************************************
> ; assign file attributes
> ;***********************************************
> fAtt = True
> fAtt_at_title = "GRIB-to-netCDF: Efficient Approach"
>
> fAtt_at_source_file = "e4oper.an.pl.t106.q.1980"
> fAtt_at_Conventions = "None"
>
> fAtt_at_creation_date = systemfunc("date")
> fileattdef( ncdf, fAtt )
>
> ;***********************************************
> ; predefine coordinate information
> ;***********************************************
> dimNames = (/"time", "lat", "lon", "level" /)
> dimSizes = (/ ntim , nlat, mlon, klev /)
> dimUnlim = (/ True , False, False, False /)
> filedimdef(ncdf, dimNames , dimSizes, dimUnlim )
> filevardef(ncdf, "time" , typeof(time), getvardims(time) )
> filevarattdef(ncdf, "time", time)
> filevardef(ncdf, "level" , typeof(level) , getvardims(level) )
> filevarattdef(ncdf, "level", level)
> filevardef(ncdf, "lat", typeof(lat), getvardims(lat) )
> filevarattdef(ncdf, "lat", lat)
> filevardef(ncdf, "lon", typeof(lon), getvardims(lon) )
> filevarattdef(ncdf, "lon", lon)
>
> ;***********************************************
> ; predefine variable sizes
> ;***********************************************
> filevardef(ncdf,"shum",typeof(shum), getvardims(shum) )
> filevarattdef(ncdf,"shum",shum)
>
> ;***********************************************
> ; terminate define mode: not necessary but for clarity
> ;***********************************************
> setfileoption(ncdf, "definemode", False)
>
> ;***********************************************
> ; write data values to predefined locations
> ; (/ .../) operator transfer values only
> ;***********************************************
> ncdf->time = (/ time /)
> ncdf->level = (/ level /)
> ncdf->lat = (/ lat /)
> ncdf->lon = (/ lon /)
>
> ncdf->shum = (/ shum /)
> end
>
>
_______________________________________________
ncl-talk mailing list
ncl-talk_at_ucar.edu
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Fri May 16 2008 - 07:34:22 MDT
This archive was generated by hypermail 2.2.0 : Mon May 19 2008 - 08:42:48 MDT