Re: addfile question

Matt,

Here is a method to check for truncated grib files. Use the Unix dd
command to get the last four bytes of the file. These bytes should
be "7777" in ascii for valid grib files, and something else if the
file was truncated.

Attached is a stand-alone shell script that you can use. The
command line usage is "./eof-check filename".

Call this with the NCL systemfunc function, parse the output, and
compare with " 7 7 7 7" in an if statement. Due to the output
format of the od command, the leading and embedded spaces in this
compare string are required.

Since grib1 and grib2 formats both use identical end sections, this
method should work for both.

Caveats:

1. Though extremely unlikely, this method will falsely approve a
truncated file that happens to have "7777" in a data array, exactly
at the break point. If this becomes a problem, there is a more
reliable but complicated method to check grib files.

2. There are variants of the "dd" command out there. You may need
to adjust the command arguments for your version.

3. Beware of extra messages from "dd" to standard error. I suspect
that they will just print harmlessly to your console. However,
standard error handling is not currently defined in the
documentation for systemfunc.

4. Some databases have grib files padded with zeros to block
boundaries. If you get any of these, you will have to adjust the
parsing accordingly.

Grib1 reference:
http://www.wmo.ch/web/www/WDM/Guides/Guide-binary-2.html

Grib2 reference:
http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc.shtml

I hope this is of some use to you.

--Dave A.
CU/CIRES Climate Diagnostics Center (CDC)
NOAA/ESRL/PSD, Climate Analysis Branch (CAB)

Matt Fearon wrote:
> Dear NCL Users:
>
> When I use addfile on a particular grib file, ie,
> f=addfile(test.grb,"r"), I receive the following error message:
>
> fatal:Premature end-of-file, file appears to be truncated
>
> I realize this is a problem with the file. However, is there an easy
> method to check for this error so my script
> will not crash. I would like to be able to skip such files. I have tried
> the following, but have been
> unsuccessful;
>
> if (any(f.eq."fatal")) then
> print("file truncated, skip to next file")
> end if
>
> Any suggestions would be greatly appreciated.
> thanks,
> Matt

#!/bin/sh
#
# Show last four bytes of any file.
# Used to check for valid terminator '7777' on grib files, among other things.
#
# 2004-dec-23 Original version by Hoop.
# 2007-jun-04 Fix for Mac OS (GNU) version of dd. Change 'iseek' to 'skip'.
# 2007-jun-05 Show exactly last four bytes.
#
FILESIZE=`ls -lL $1 | awk '{print $5}'`
SKIPCNT=`expr $FILESIZE - 4`
dd if=$1 ibs=1 skip=$SKIPCNT | od -c

_______________________________________________
ncl-talk mailing list
ncl-talk_at_ucar.edu
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Tue Jun 05 2007 - 14:37:37 MDT