Re: Adressing a variable > 2 Gb < 4 Gb

Hi Christian,

The current plan is to have this implemented for the 5.2.0 release
(along with a host of other new features). The timeline for that
release has not been firmly fixed yet, but is at least several months
out.

Rick Brownrigg

On Sep 30, 2009, at 8:54 AM, Christian Pagé wrote:

> Hello all,
>
> In March 2008, I have asked that question, about the possibility to
> have greater than 2 Gb variable, but less than 4 Gb (see emails
> below).
>
> I tried again today with NCL 5.1.1
> ncl_ncarg-5.1.1.Linux_x86_64_gcc412.tar.gz, and it still doesn't
> work. Any plan yet for an NCL release which will support it?
>
> I was trying to open a variable stored in 87 files of 28 Mb each.
>
> Here is the valgrind output:
>
>
> ==11968== Source and destination overlap in memcpy(0xE748EE0,
> 0xE749030, 27976520)
> ==11968== at 0x4907456: memcpy (mc_replace_strmem.c:402)
> ==11968== by 0x5E0382: CallLIST_READ_FILEVAR_OP (in /contrex/
> local64/ncl/bin/ncl)
> ==11968== by 0x5E09C4: _NclExecute (in /contrex/local64/ncl/bin/
> ncl)
> ==11968== by 0x5E6A8E: _NclFuncCallOp (in /contrex/local64/ncl/
> bin/ncl)
> ==11968== by 0x5DABAE: CallFUNC_CALL_OP (in /contrex/local64/ncl/
> bin/ncl)
> ==11968== by 0x5E0884: _NclExecute (in /contrex/local64/ncl/bin/
> ncl)
> ==11968== by 0x51FC68: yyparse (in /contrex/local64/ncl/bin/ncl)
> ==11968== by 0x51B4CD: main (in /contrex/local64/ncl/bin/ncl)
> ==11968==
> ==11968== Invalid write of size 1
> ==11968== at 0x49074FC: memcpy (mc_replace_strmem.c:402)
> ==11968== by 0x5E0382: CallLIST_READ_FILEVAR_OP (in /contrex/
> local64/ncl/bin/ncl)
> ==11968== by 0x5E09C4: _NclExecute (in /contrex/local64/ncl/bin/
> ncl)
> ==11968== by 0x5E6A8E: _NclFuncCallOp (in /contrex/local64/ncl/
> bin/ncl)
> ==11968== by 0x5DABAE: CallFUNC_CALL_OP (in /contrex/local64/ncl/
> bin/ncl)
> ==11968== by 0x5E0884: _NclExecute (in /contrex/local64/ncl/bin/
> ncl)
> ==11968== by 0x51FC68: yyparse (in /contrex/local64/ncl/bin/ncl)
> ==11968== by 0x51B4CD: main (in /contrex/local64/ncl/bin/ncl)
> ==11968== Address 0xe748ee0 is 0 bytes after a block of size
> 28,053,168 alloc'd
> ==11968== at 0x4904E57: malloc (vg_replace_malloc.c:207)
> ==11968== by 0x568182: NclMalloc (Memory.c:57)
> ==11968== by 0x5754AB: MyFileReadVarValue (in /contrex/local64/
> ncl/bin/ncl)
> ==11968== by 0x578095: FileReadVar (in /contrex/local64/ncl/bin/
> ncl)
> ==11968== by 0x554444: _NclFileReadVar (in /contrex/local64/ncl/
> bin/ncl)
> ==11968== by 0x5DFB01: CallLIST_READ_FILEVAR_OP (in /contrex/
> local64/ncl/bin/ncl)
> ==11968== by 0x5E09C4: _NclExecute (in /contrex/local64/ncl/bin/
> ncl)
> ==11968== by 0x5E6A8E: _NclFuncCallOp (in /contrex/local64/ncl/
> bin/ncl)
> ==11968== by 0x5DABAE: CallFUNC_CALL_OP (in /contrex/local64/ncl/
> bin/ncl)
> ==11968== by 0x5E0884: _NclExecute (in /contrex/local64/ncl/bin/
> ncl)
> ==11968== by 0x51FC68: yyparse (in /contrex/local64/ncl/bin/ncl)
> ==11968== by 0x51B4CD: main (in /contrex/local64/ncl/bin/ncl)
>
>
> Best regards,
>
> Christian Page
>
> =======================
>
>
> Hello Rick,
> I would be quite interested in testing it ! As soon as you have a
> working
> version, please send it to me and I will test it.
> The servers I am working on are :
> Linux 2.6.22.14-72.fc6 #1 SMP Wed Nov 21 14:10:25 EST 2007 x86_64
> x86_64
> x86_64 GNU/Linux
> for the 64-bit linux
> and :
> Linux 2.6.20-1.2320.fc5smp #1 SMP Tue Jun 12 19:40:16 EDT 2007 i686
> i686
> i386 GNU/Linux
> for 32-bit linux.
> Christian
> 2008/3/17, Rick Grubin <grubin_at_ucar.edu>:
> >
> > Hello Christian
> >
> >
> > > I would need to address a variable past the 2 Gb memory limit. The
> > > variable is read off from addfiles using addfiles_GetVar on
> several
> > > separate small NetCDF classic files. Is it possible to achieve
> > > this ? I have both 32-bit and 64-bit Linux machines I can use.
> >
> >
> > At the moment, NCL does not handle variables > 2Gb, regardless of
> the
> > host you are on. This is a limitation internal to NCL, based on
> > its current implementation.
> >
> > That said, we have been working on upgrading NCL to be able to
> > address a variable > 2Gb. We have made good progress and hope to
> > present to users a version for testing very soon. Would you be
> > interested in helping us test?
> >
> > ----
> >
> > Rick Grubin NCAR/TDD/CISL/VETS
> > Visualization + Enabling Technologies
> > grubin_at_ucar.edu 303.497.1832
> >
> >
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Wed Sep 30 2009 - 09:52:30 MDT