Re: ESMF regrid

From: Mary Haley <haley_at_nyahnyahspammersnyahnyah>
Date: Tue Mar 19 2013 - 08:05:06 MDT

Hi Noel,

I see what the problem is, and I'm not sure yet how to fix it.

The issue is in order to do ESMF regridding, we need to write an intermediate NetCDF SCRIP file. This file will have two double variables of size grid_size, which is equal to nlat * nlon.

In your case:

   nlat/nlon = 16,800/43,200
   grid_size = nlat*lon = 725760000
   variable to be written to the file = 725760000*8 = 5,806,080,000 bytes

This is > 4 GB. This violates a NetCDF constraint that individual variables be less than 4 GB.

My first thought is that if you can create a NetCDF-4 file, then you won't be limited by this constraint. The problem is that the ESMF regridding tool we use appears to have a constraint that the intermediate NetCDF files be NetCDF-3 (classic or 64-bit offset) files, and not NetCDF-4.

I will check into this and get back to you.

--Mary

On Mar 7, 2013, at 10:27 AM, Noel Aloysius wrote:

> Hi Mary,
>
> The data file is uploaded to the ftp server - ftp://ftp.cgd.ucar.edu/incoming/MOD17A3_Science_GPP_2001.zip.
>
> The script is attached.
>
> I want to regrid ftom 1km to 0.5x0.5deg lat/lon.
>
> Thanks,
> Noel
>
>
> Noel Aloysius
>
>
> On Thu, Mar 7, 2013 at 9:56 AM, Mary Haley <haley@ucar.edu> wrote:
> Hi Noel,
>
> Can you provide me with your script and data? It looks like there's some problem creating the source description file.
> The error makes me think that the file is too large, but the SrcLargeFile should have handled this.
>
> --Mary
>
> On Mar 5, 2013, at 4:34 PM, Noel Aloysius wrote:
>
> > Hi Mary,
> >
> > I am using NCL 6.1.0.
> >
> > I modified the script as per you suggestion, but still get the following error message.
> >
> > It works when I select a smaller region though.
> >
> > ncendef: ncid 131072: NetCDF: One or more variable sizes violate format constraints
> > fatal:NetCDF: Operation not allowed in define mode: error attempting to write variable (grid_dims) to file (/home/fas/saiers/nra23/scratch/GPP_NPP_MODIS/GPP_src_SCRIP.nc)
> > fatal:["Execute.c":7743]:Execute: Error occurred at or near line 1168 in file $NCARG_ROOT/lib/ncarg/nclscripts/esmf/ESMF_regridding.ncl
> > fatal:["Execute.c":7743]:Execute: Error occurred at or near line 1277 in file $NCARG_ROOT/lib/ncarg/nclscripts/esmf/ESMF_regridding.ncl
> > fatal:["Execute.c":7743]:Execute: Error occurred at or near line 2976 in file $NCARG_ROOT/lib/ncarg/nclscripts/esmf/ESMF_regridding.ncl
> > fatal:["Execute.c":7743]:Execute: Error occurred at or near line 3089 in file $NCARG_ROOT/lib/ncarg/nclscripts/esmf/ESMF_regridding.ncl
> > fatal:["Execute.c":7741]:Execute: Error occurred at or near line 90
> >
> > Thanks,
> > Noel
> >
> >
> >
> > On Tue, Mar 5, 2013 at 5:25 PM, Mary Haley <haley@ucar.edu> wrote:
> > Hi Noel,
> >
> > What version of NCL are you using? It looks like a large file issue, so I think this should fix the problem:
> >
> > Opt@SrcLargeFile = True
> > Opt@DstLargeFile = True
> >
> > These are the defaults for Version 6.1.2.
> >
> > --Mary
> >
> > On Mar 5, 2013, at 1:01 PM, Noel Aloysius wrote:
> >
> > > Hi NCL-talk,
> > >
> > > I want to regrid a high-resolution (~1km) MODIS GPP to a coarse resolution (1x1) grid using ESMF_regrid. The script is appended below.
> > > I get the following error messages,
> > >
> > > ncendef: ncid 131072: NetCDF: One or more variable sizes violate format constraints
> > > fatal:NetCDF: Operation not allowed in define mode: error attempting to write variable (grid_dims) to file (/home/fas/scratch/GPP_NPP_MODIS/src_SCRIP.nc)
> > > fatal:["Execute.c":7743]:Execute: Error occurred at or near line 1168 in file $NCARG_ROOT/lib/ncarg/nclscripts/esmf/ESMF_regridding.ncl
> > > fatal:["Execute.c":7743]:Execute: Error occurred at or near line 1277 in file $NCARG_ROOT/lib/ncarg/nclscripts/esmf/ESMF_regridding.ncl
> > > fatal:["Execute.c":7743]:Execute: Error occurred at or near line 2976 in file $NCARG_ROOT/lib/ncarg/nclscripts/esmf/ESMF_regridding.ncl
> > > fatal:["Execute.c":7743]:Execute: Error occurred at or near line 3089 in file $NCARG_ROOT/lib/ncarg/nclscripts/esmf/ESMF_regridding.ncl
> > > fatal:["Execute.c":7741]:Execute: Error occurred at or near line 27
> > >
> > > Thanks in advance for your help to resolve this issue,
> > > Noel
> > >
> > >
> > > ;**************************************************************************
> > > srcDir = "~/"
> > > srcFile = "MOD17A3_Science_GPP_2000.nc"
> > > srcFileName = str_concat((/srcDir,srcFile/))
> > >
> > > sfile = addfile(srcFileName,"r") ; source grid
> > > gpp = int2flt (sfile->MOD17A3_Science_GPP_2000)
> > > gpp = gpp(::-1,:) ; South -> North
> > >
> > > Opt = True
> > >
> > > Opt@SrcFileName = srcDir + "src_SCRIP.nc"
> > > Opt@DstFileName = srcDir + "dst_SCRIP.nc"
> > > Opt@WgtFileName = srcDir + "GPP1km_2_halfddeg.nc"
> > > Opt@ForceOverwrite = True
> > >
> > > Opt@DstGridType = "1x1"
> > > Opt@DstLLCorner = (/ -59.5, -179.5/)
> > > Opt@DstURCorner = (/ 79.5, 179.5/)
> > >
> > > gpp_regrid = ESMF_regrid(gpp,Opt)
> > > ;**************************************************************************
> > >
> > > printVarSummary(gpp),
> > >
> > > Variable: gpp
> > > Type: float
> > > Total Size: 2903040000 bytes
> > > 725760000 values
> > > Number of Dimensions: 2
> > > Dimensions and sizes: [lat | 16800] x [lon | 43200]
> > > Coordinates:
> > > lat: [-59.99583333334029..79.99583333333334]
> > > lon: [-179.9958333333333..179.9958333333346]
> > > Number Of Attributes: 6
> > > _FillValue : 1e+20
> > > long_name : MOD17A3_Science_GPP_2000
> > > esri_pe_string : GEOGCS["GCS_WGS_1984",DATUM["D_WGS_1984",SPHEROID["WGS_1984",6378137.0,298.257223563]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]]
> > > coordinates : lon lat
> > > units : Degree
> > > missing_value : 0
> > >
> > > ncl 12> print(min(gpp))
> > > (0) 1
> > > ncl 13> print(max(gpp))
> > > (0) 65535
> > >
> > >
> > > _______________________________________________
> > > ncl-talk mailing list
> > > List instructions, subscriber options, unsubscribe:
> > > http://mailman.ucar.edu/mailman/listinfo/ncl-talk
> >
> >
> > _______________________________________________
> > ncl-talk mailing list
> > List instructions, subscriber options, unsubscribe:
> > http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
> <ESMF_regrid_MODIS_Annual_GPP_for_NCL_talk.ncl>_______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Tue Mar 19 08:05:17 2013

This archive was generated by hypermail 2.1.8 : Tue Mar 19 2013 - 16:31:18 MDT