Re: addfile question

Hi Dave,
Thanks for your email. This looks extremely handy. I implemented
something similar using an older version of wgrib I got from w.
ebisuzaki. However, your method may more robust, and I may switch to it.
Maybe I am crazy, but the function systemfunc seems to produce
inconsistent results. It is frustrating me. I am using tcsh as my shell
- maybe that's the problem, and I should just use bash.
Matt

Dave Allured wrote:
> Matt,
>
> Here is a method to check for truncated grib files. Use the Unix dd
> command to get the last four bytes of the file. These bytes should be
> "7777" in ascii for valid grib files, and something else if the file
> was truncated.
>
> Attached is a stand-alone shell script that you can use. The command
> line usage is "./eof-check filename".
>
> Call this with the NCL systemfunc function, parse the output, and
> compare with " 7 7 7 7" in an if statement. Due to the output
> format of the od command, the leading and embedded spaces in this
> compare string are required.
>
> Since grib1 and grib2 formats both use identical end sections, this
> method should work for both.
>
> Caveats:
>
> 1. Though extremely unlikely, this method will falsely approve a
> truncated file that happens to have "7777" in a data array, exactly at
> the break point. If this becomes a problem, there is a more reliable
> but complicated method to check grib files.
>
> 2. There are variants of the "dd" command out there. You may need to
> adjust the command arguments for your version.
>
> 3. Beware of extra messages from "dd" to standard error. I suspect
> that they will just print harmlessly to your console. However,
> standard error handling is not currently defined in the documentation
> for systemfunc.
>
> 4. Some databases have grib files padded with zeros to block
> boundaries. If you get any of these, you will have to adjust the
> parsing accordingly.
>
> Grib1 reference:
> http://www.wmo.ch/web/www/WDM/Guides/Guide-binary-2.html
>
> Grib2 reference:
> http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc.shtml
>
> I hope this is of some use to you.
>
> --Dave A.
> CU/CIRES Climate Diagnostics Center (CDC)
> NOAA/ESRL/PSD, Climate Analysis Branch (CAB)
>
> Matt Fearon wrote:
>> Dear NCL Users:
>>
>> When I use addfile on a particular grib file, ie,
>> f=addfile(test.grb,"r"), I receive the following error message:
>>
>> fatal:Premature end-of-file, file appears to be truncated
>>
>> I realize this is a problem with the file. However, is there an easy
>> method to check for this error so my script
>> will not crash. I would like to be able to skip such files. I have
>> tried the following, but have been
>> unsuccessful;
>>
>> if (any(f.eq."fatal")) then
>> print("file truncated, skip to next file")
>> end if
>>
>> Any suggestions would be greatly appreciated.
>> thanks,
>> Matt
> ------------------------------------------------------------------------
>
> #!/bin/sh
> #
> # Show last four bytes of any file.
> # Used to check for valid terminator '7777' on grib files, among other things.
> #
> # 2004-dec-23 Original version by Hoop.
> # 2007-jun-04 Fix for Mac OS (GNU) version of dd. Change 'iseek' to 'skip'.
> # 2007-jun-05 Show exactly last four bytes.
> #
> FILESIZE=`ls -lL $1 | awk '{print $5}'`
> SKIPCNT=`expr $FILESIZE - 4`
> dd if=$1 ibs=1 skip=$SKIPCNT | od -c
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk_at_ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>

_______________________________________________
ncl-talk mailing list
ncl-talk_at_ucar.edu
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Tue Jun 05 2007 - 15:24:16 MDT