Re: ESMF regrid problem

Hi Will,

I had to do a little digging to see where we are with this issue (I have
filed a new ticket on this, NCL-1989).

In order to support large variables that are > 4 GB, you have to write them
to NetCDF-4 file (the pure one, not the classic one). This means that
ESMF_RegridWeightGen needs to be able to read these files.

It appears that the latest version of ESMF_RegridWeightGen (6.3.0r) does
not support this yet, and I have question into the developers about this.

It does appear that you can *write* the weights file that
ESMF_RegridWeightGen creates to a NetCDF-4 file, but this isn't going to
help you much.

Are your lat/lon dimensions causing you to have a variable that is > 4 GB,
or do you have other dimensions that are contributing to a large variable?

I recall having one user get around a similar problem by regridding subsets
of the variable and stitching them together later. His situation was a
little different because his variable was already naturally subsetted into
individual files.

If your grid isn't too irregular, then a step-by-step method like this
might work for you.

--Mary



On Tue, Jul 8, 2014 at 12:40 AM, Will Hobbs <Will.Hobbs@utas.edu.au> wrote:

> Mary et al,
>
> I am having problems running out of memory using the ESMF_regrid
> function. The problem is basically the same as reported last year by a user
> on March 19th, (Mary's reply is copied below), but I'm just checking to see
> if there was a resolution?
>
> Many thanks
>
> Will
>
>
> From: Mary Haley <haley_at_nyahnyahspammersnyahnyah
> <haley_at_nyahnyahspammersnyahnyah?Subject=Re:%20ESMF%20regrid>>
> Date: Tue Mar 19 2013 - 08:05:06 MDT
>
> Hi Noel,
>
> I see what the problem is, and I'm not sure yet how to fix it.
>
> The issue is in order to do ESMF regridding, we need to write an
> intermediate NetCDF SCRIP file. This file will have two double variables of
> size grid_size, which is equal to nlat * nlon.
>
> In your case:
>
> nlat/nlon = 16,800/43,200
> grid_size = nlat*lon = 725760000
> variable to be written to the file = 725760000*8 = 5,806,080,000 bytes
>
> This is > 4 GB. This violates a NetCDF constraint that individual
> variables be less than 4 GB.
>
> My first thought is that if you can create a NetCDF-4 file, then you won't
> be limited by this constraint. The problem is that the ESMF regridding tool
> we use appears to have a constraint that the intermediate NetCDF files be
> NetCDF-3 (classic or 64-bit offset) files, and not NetCDF-4.
>
> I will check into this and get back to you.
>
> --Mary
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>

Received on Tue Jul 08 10:01:59 2014