Re: problems when reading binary direct access big endian files which are large

Jatin,

It looks like this issue is related to the 2G limit
(As you data is just above 2G ~ 2147483648).

We are checking to see if we can figure out a way to
write a data block larger than 2G.

Regards,

Wei

huangwei@ucar.edu
VETS/CISL
National Center for Atmospheric Research
P.O. Box 3000 (1850 Table Mesa Dr.)
Boulder, CO 80307-3000 USA
(303) 497-8924

On Jan 28, 2014, at 11:35 PM, Jatin Kala <jatin.kala.jk@gmail.com> wrote:

> Hi NCL'ers,
>
> We need to read a binary big endian direct-acess file, do stuff with the data and re-write the file to binary big endian direct access again.
> We are having trouble when re-writing, with this error:
> fatal:fbindirread: The size implied by the dimension array and record number is greater that the size of the file, can't continue
>
> Below is a simple script to illustrate the problem. When the array size is small, e.g, dx and dy = 100, the code works fine, no problem. When the array is large as in this example, we get the error.
>
> begin
> ;--------------------- Inputs -------------------------
> ; define size array
> dx = 15000
> dy = 36000
> ;-----------------------------------------------------
> x = new((/dx,dy/),"float")
> x(:,:) = 1.0
> ; write x to direct access big endian
> setfileoption("bin","WriteByteOrder","BigEndian")
> system("rm -f test.bin")
> fbindirwrite("test.bin",x)
> delete(x)
> ; re-read the file
> setfileoption("bin","ReadByteOrder","BigEndian")
> x = fbindirread("test.bin",0,(/dx,dy/),"float")
> ;------------------------------------------------------
> end ; this ends begin
>
>
> Hence, this somehow seems related to file size. There is a "large file" type option for netcdf files, but not for binary files it seems?
> I have already googled the error, and setting RecordMarkerSize will not fix the problem as it applies to sequential files….
>
> Also, when dy and dx = 100, the file size makes sense:
> [z3381484@monsoon ncl-scripts-general]$ ls -l test.bin
> -rwx------. 1 z3381484 unsw 40000 Jan 29 17:21 test.bin
> I.e, 100*100*4 = 40000
>
> When dx = 15000, and dy = 36000, the size should be 15000*36000*4 = 2160000000, but instead it is:
> [z3381484@monsoon ncl-scripts-general]$ ls -l test.bin
> -rwx------. 1 z3381484 unsw 2147479552 Jan 29 17:31 test.bin
>
> The machine i am running on is 64 bit Linux:
> [z3381484@monsoon ncl-scripts-general]$ uname -a
> Linux monsoon.ccrc.unsw.edu.au 2.6.32-358.23.2.el6.x86_64 #1 SMP Wed Oct 16 18:37:12 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
>
> version of NCL is:
> [z3381484@monsoon ncl-scripts-general]$ ncl -V
> 6.1.2
>
> Cheers,
> Jatin.
>
>
>
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Wed Jan 29 11:30:32 2014