Re: ESMF regrid problem

Hi Will,

I talked to the ESMF folks about the NetCDF-4 issues. I was able to create
the intermediate source and destination files as NetCDF-4 files, but then
the ESMF_RegridWeightGen tool couldn't read them back in. This tool
internally uses the Fortran 90 tool to read the NetCDF-4 files, and this
interface doesn't yet support the NetCDF-4 "string" type. The intermediate
files have NetCDF-4 strings in them, so hence the failure.

I am able to work-around this problem by forcing any file variable
attributes to be written as character arrays. This enabled me to still
create the NetCDF-4 files and then be able to read them in with the
ESMF_RegridWeightGen tool. If you want to try a modified version of
ESMF_regridding.ncl with this work-around, let me know.

Meanwhile, with regard to your large time array, is the lat/lon grid the
same for every timestep? If so, you can create the weights file based on
just one time step, and then use ESMF_regrid_with_weights to regrid the
rest of the time steps. However, from the error message you showed me, I'm
not sure this was the problem.

If you are trying to write a variable that is larger than 4 GB, then you
must use a NetCDF-4 file, and thus you need to have this line before you
call addfile in "create" mode:

  setfileoption("nc","format","NetCDF4")

I have to step out for a bit, but if you want to pursue this further, we
can take it offline briefly. It would also help if you could provide a
script (and data, of course, or at least information about the data, if
it's too large to transfer).

Thanks,

--Mary



On Tue, Jul 8, 2014 at 5:18 PM, Will Hobbs <Will.Hobbs@utas.edu.au> wrote:

> Hi Mary
>
> I suspect that the file size issue is related to the time dimension
> rather than just the grid size, so I could try and use a time loop to break
> things up a bit.
>
> I was reluctant to do this since the script needs to run on all 45
> control runs from the CMIP5 models, so it's already a big, slow chunk of
> processing. Obviously it also needs to be a fairly 'general' regrid, with
> lots of exception handling, since there are so many different grids to deal
> with.
>
> Will
>
> From: Mary Haley <haley@ucar.edu>
> Date: Wednesday, 9 July 2014 8:01 AM
> To: Will Hobbs <will.hobbs@utas.edu.au>
> Cc: "ncl-talk@ucar.edu" <ncl-talk@ucar.edu>
> Subject: Re: ESMF regrid problem
>
> Hi Will,
>
> I had to do a little digging to see where we are with this issue (I have
> filed a new ticket on this, NCL-1989).
>
> In order to support large variables that are > 4 GB, you have to write
> them to NetCDF-4 file (the pure one, not the classic one). This means that
> ESMF_RegridWeightGen needs to be able to read these files.
>
> It appears that the latest version of ESMF_RegridWeightGen (6.3.0r) does
> not support this yet, and I have question into the developers about this.
>
> It does appear that you can *write* the weights file that
> ESMF_RegridWeightGen creates to a NetCDF-4 file, but this isn't going to
> help you much.
>
> Are your lat/lon dimensions causing you to have a variable that is > 4
> GB, or do you have other dimensions that are contributing to a large
> variable?
>
> I recall having one user get around a similar problem by regridding
> subsets of the variable and stitching them together later. His situation
> was a little different because his variable was already naturally subsetted
> into individual files.
>
> If your grid isn't too irregular, then a step-by-step method like this
> might work for you.
>
> --Mary
>
>
>
> On Tue, Jul 8, 2014 at 12:40 AM, Will Hobbs <Will.Hobbs@utas.edu.au>
> wrote:
>
>> Mary et al,
>>
>> I am having problems running out of memory using the ESMF_regrid
>> function. The problem is basically the same as reported last year by a user
>> on March 19th, (Mary's reply is copied below), but I'm just checking to see
>> if there was a resolution?
>>
>> Many thanks
>>
>> Will
>>
>>
>> From: Mary Haley <haley_at_nyahnyahspammersnyahnyah
>> <haley_at_nyahnyahspammersnyahnyah?Subject=Re:%20ESMF%20regrid>>
>> Date: Tue Mar 19 2013 - 08:05:06 MDT
>>
>> Hi Noel,
>>
>> I see what the problem is, and I'm not sure yet how to fix it.
>>
>> The issue is in order to do ESMF regridding, we need to write an
>> intermediate NetCDF SCRIP file. This file will have two double variables of
>> size grid_size, which is equal to nlat * nlon.
>>
>> In your case:
>>
>> nlat/nlon = 16,800/43,200
>> grid_size = nlat*lon = 725760000
>> variable to be written to the file = 725760000*8 = 5,806,080,000 bytes
>>
>> This is > 4 GB. This violates a NetCDF constraint that individual
>> variables be less than 4 GB.
>>
>> My first thought is that if you can create a NetCDF-4 file, then you
>> won't be limited by this constraint. The problem is that the ESMF
>> regridding tool we use appears to have a constraint that the intermediate
>> NetCDF files be NetCDF-3 (classic or 64-bit offset) files, and not
>> NetCDF-4.
>>
>> I will check into this and get back to you.
>>
>> --Mary
>>
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>

Received on Thu Jul 10 03:18:33 2014