Re: problems when reading binary direct access big endian files which are large

Hi Wei,
Thanks for your quick reply.
1. Nopes, the sys admin would have used the binary install, and from what i've heard, building NCL from scratch is rather painful!
2. Will not work either, as the file is read later by a land surface model, and it has to be a single direct access binary big endian file.

The solution for me i think is to use Fortran with ifort and not use NCL at all for this task. I've tested, and writing binary files > 2gb with intel's compilers is Not a problem and works fine. So, good-old Fortran it is!

It would be good to point out this limitation on the NCL web-page where you guys describe binary file I/O in NCL.
Cheers,
Jatin.

On 30/01/2014, at 7:53 AM, Wei Huang wrote:

> Jatin,
>
> Apparently, this is a gcc (or more precisely a gfortran) issue, which internally,
> each written block has a limit of 2G.
> There are several ways to work around this issue.
>
> 1. If you generated NCL executable yourself from source code,
> you may try to re-generate it using icc, or pgcc.
>
> 2. You can write your variable to parts, and read them in separately.
> A sample script like below should work:
>
> ;define size array
> dx = 15000
> dy = 36000
> ;-----------------------------------------------------
> x = new((/dx,dy/),"float")
> x = 1.0
> ;write x to direct access big endian
> setfileoption("bin","WriteByteOrder","BigEndian")
> system("rm -f test.bin")
> ;fbindirwrite("test.bin",x)
> fbinrecwrite("test.bin",-1,x(0:dx/2-1,:))
> fbinrecwrite("test.bin",-1,x(dx/2:,:))
> ;delete(x)
>
> ;-----------------------------------------------------
> ;re-read the file
> x = 0.0
> setfileoption("bin","ReadByteOrder","BigEndian")
> x(0:dx/2-1,:) = fbinrecread("test.bin",0,(/(dx/2),dy/),"float")
> x(dx/2:,:) = fbinrecread("test.bin",1,(/(dx/2),dy/),"float")
> printVarSummary(x)
> print("x(0, 0) = " + x(0, 0))
> print("x(dx/2, 0) = " + x(dx/2, 0))
>
>
> If you are curious about the source of this issue, please take a look at:
> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44292
>
> Unfortunately, they did not fix this issue in the new release 4.8.2.
>
> Thanks,
>
> Wei
>
> huangwei@ucar.edu
> VETS/CISL
> National Center for Atmospheric Research
> P.O. Box 3000 (1850 Table Mesa Dr.)
> Boulder, CO 80307-3000 USA
> (303) 497-8924
>
>
>
>
>
>
> On Jan 29, 2014, at 11:30 AM, Wei Huang <huangwei@ucar.edu> wrote:
>
>> Jatin,
>>
>> It looks like this issue is related to the 2G limit
>> (As you data is just above 2G ~ 2147483648).
>>
>> We are checking to see if we can figure out a way to
>> write a data block larger than 2G.
>>
>> Regards,
>>
>> Wei
>>
>> huangwei@ucar.edu
>> VETS/CISL
>> National Center for Atmospheric Research
>> P.O. Box 3000 (1850 Table Mesa Dr.)
>> Boulder, CO 80307-3000 USA
>> (303) 497-8924
>>
>>
>>
>>
>>
>>
>> On Jan 28, 2014, at 11:35 PM, Jatin Kala <jatin.kala.jk@gmail.com> wrote:
>>
>>> Hi NCL'ers,
>>>
>>> We need to read a binary big endian direct-acess file, do stuff with the data and re-write the file to binary big endian direct access again.
>>> We are having trouble when re-writing, with this error:
>>> fatal:fbindirread: The size implied by the dimension array and record number is greater that the size of the file, can't continue
>>>
>>> Below is a simple script to illustrate the problem. When the array size is small, e.g, dx and dy = 100, the code works fine, no problem. When the array is large as in this example, we get the error.
>>>
>>> begin
>>> ;--------------------- Inputs -------------------------
>>> ; define size array
>>> dx = 15000
>>> dy = 36000
>>> ;-----------------------------------------------------
>>> x = new((/dx,dy/),"float")
>>> x(:,:) = 1.0
>>> ; write x to direct access big endian
>>> setfileoption("bin","WriteByteOrder","BigEndian")
>>> system("rm -f test.bin")
>>> fbindirwrite("test.bin",x)
>>> delete(x)
>>> ; re-read the file
>>> setfileoption("bin","ReadByteOrder","BigEndian")
>>> x = fbindirread("test.bin",0,(/dx,dy/),"float")
>>> ;------------------------------------------------------
>>> end ; this ends begin
>>>
>>>
>>> Hence, this somehow seems related to file size. There is a "large file" type option for netcdf files, but not for binary files it seems?
>>> I have already googled the error, and setting RecordMarkerSize will not fix the problem as it applies to sequential files….
>>>
>>> Also, when dy and dx = 100, the file size makes sense:
>>> [z3381484@monsoon ncl-scripts-general]$ ls -l test.bin
>>> -rwx------. 1 z3381484 unsw 40000 Jan 29 17:21 test.bin
>>> I.e, 100*100*4 = 40000
>>>
>>> When dx = 15000, and dy = 36000, the size should be 15000*36000*4 = 2160000000, but instead it is:
>>> [z3381484@monsoon ncl-scripts-general]$ ls -l test.bin
>>> -rwx------. 1 z3381484 unsw 2147479552 Jan 29 17:31 test.bin
>>>
>>> The machine i am running on is 64 bit Linux:
>>> [z3381484@monsoon ncl-scripts-general]$ uname -a
>>> Linux monsoon.ccrc.unsw.edu.au 2.6.32-358.23.2.el6.x86_64 #1 SMP Wed Oct 16 18:37:12 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
>>>
>>> version of NCL is:
>>> [z3381484@monsoon ncl-scripts-general]$ ncl -V
>>> 6.1.2
>>>
>>> Cheers,
>>> Jatin.
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> ncl-talk mailing list
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Wed Jan 29 15:40:54 2014