Re: problems when reading binary direct access big endian files which are large

Jatin,

Apparently, this is a gcc (or more precisely a gfortran) issue, which internally,
each written block has a limit of 2G.
There are several ways to work around this issue.

1. If you generated NCL executable yourself from source code,
        you may try to re-generate it using icc, or pgcc.

2. You can write your variable to parts, and read them in separately.
        A sample script like below should work:

;define size array
 dx = 15000
 dy = 36000
;-----------------------------------------------------
 x = new((/dx,dy/),"float")
 x = 1.0
;write x to direct access big endian
 setfileoption("bin","WriteByteOrder","BigEndian")
 system("rm -f test.bin")
;fbindirwrite("test.bin",x)
 fbinrecwrite("test.bin",-1,x(0:dx/2-1,:))
 fbinrecwrite("test.bin",-1,x(dx/2:,:))
;delete(x)

;-----------------------------------------------------
;re-read the file
 x = 0.0
 setfileoption("bin","ReadByteOrder","BigEndian")
 x(0:dx/2-1,:) = fbinrecread("test.bin",0,(/(dx/2),dy/),"float")
 x(dx/2:,:) = fbinrecread("test.bin",1,(/(dx/2),dy/),"float")
 printVarSummary(x)
 print("x(0, 0) = " + x(0, 0))
 print("x(dx/2, 0) = " + x(dx/2, 0))

If you are curious about the source of this issue, please take a look at:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44292

Unfortunately, they did not fix this issue in the new release 4.8.2.

Thanks,

Wei

huangwei@ucar.edu
VETS/CISL
National Center for Atmospheric Research
P.O. Box 3000 (1850 Table Mesa Dr.)
Boulder, CO 80307-3000 USA
(303) 497-8924

On Jan 29, 2014, at 11:30 AM, Wei Huang <huangwei@ucar.edu> wrote:

> Jatin,
>
> It looks like this issue is related to the 2G limit
> (As you data is just above 2G ~ 2147483648).
>
> We are checking to see if we can figure out a way to
> write a data block larger than 2G.
>
> Regards,
>
> Wei
>
> huangwei@ucar.edu
> VETS/CISL
> National Center for Atmospheric Research
> P.O. Box 3000 (1850 Table Mesa Dr.)
> Boulder, CO 80307-3000 USA
> (303) 497-8924
>
>
>
>
>
>
> On Jan 28, 2014, at 11:35 PM, Jatin Kala <jatin.kala.jk@gmail.com> wrote:
>
>> Hi NCL'ers,
>>
>> We need to read a binary big endian direct-acess file, do stuff with the data and re-write the file to binary big endian direct access again.
>> We are having trouble when re-writing, with this error:
>> fatal:fbindirread: The size implied by the dimension array and record number is greater that the size of the file, can't continue
>>
>> Below is a simple script to illustrate the problem. When the array size is small, e.g, dx and dy = 100, the code works fine, no problem. When the array is large as in this example, we get the error.
>>
>> begin
>> ;--------------------- Inputs -------------------------
>> ; define size array
>> dx = 15000
>> dy = 36000
>> ;-----------------------------------------------------
>> x = new((/dx,dy/),"float")
>> x(:,:) = 1.0
>> ; write x to direct access big endian
>> setfileoption("bin","WriteByteOrder","BigEndian")
>> system("rm -f test.bin")
>> fbindirwrite("test.bin",x)
>> delete(x)
>> ; re-read the file
>> setfileoption("bin","ReadByteOrder","BigEndian")
>> x = fbindirread("test.bin",0,(/dx,dy/),"float")
>> ;------------------------------------------------------
>> end ; this ends begin
>>
>>
>> Hence, this somehow seems related to file size. There is a "large file" type option for netcdf files, but not for binary files it seems?
>> I have already googled the error, and setting RecordMarkerSize will not fix the problem as it applies to sequential files….
>>
>> Also, when dy and dx = 100, the file size makes sense:
>> [z3381484@monsoon ncl-scripts-general]$ ls -l test.bin
>> -rwx------. 1 z3381484 unsw 40000 Jan 29 17:21 test.bin
>> I.e, 100*100*4 = 40000
>>
>> When dx = 15000, and dy = 36000, the size should be 15000*36000*4 = 2160000000, but instead it is:
>> [z3381484@monsoon ncl-scripts-general]$ ls -l test.bin
>> -rwx------. 1 z3381484 unsw 2147479552 Jan 29 17:31 test.bin
>>
>> The machine i am running on is 64 bit Linux:
>> [z3381484@monsoon ncl-scripts-general]$ uname -a
>> Linux monsoon.ccrc.unsw.edu.au 2.6.32-358.23.2.el6.x86_64 #1 SMP Wed Oct 16 18:37:12 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
>>
>> version of NCL is:
>> [z3381484@monsoon ncl-scripts-general]$ ncl -V
>> 6.1.2
>>
>> Cheers,
>> Jatin.
>>
>>
>>
>>
>>
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Wed Jan 29 13:53:58 2014