Re: addfile question

Hmmm. I use tcsh as well, but the script invokes plain ole sh (line
1). I doubt that would matter.

I never tried using the eof-check script from systemfunc. But this
reminds me of something related to caveat #3. Standard out and
Standard error are asynchronous, which means that when merged,
messages are not necessarily in the order that they were issued by
the program! Because a race is created, the order can vary from run
to run with identical input! This would perhaps explain your
"inconsistent results".

There are various ways to split standard out from standard error.
I'm not versatile on that topic. For a crude test, add a redirect
to file within your systemfunc command lne, e.g. "./eof-check
filename > check.txt". This should send only standard out to the
file, and only standard error to the systemfunc result array. Then
use NCL asciiread ("check.txt"), then parse the 7 7 7 7 out of the
first line. See if that works better.

--Dave

Matt Fearon wrote:
> Hi Dave,
> Thanks for your email. This looks extremely handy. I implemented
> something similar using an older version of wgrib I got from w.
> ebisuzaki. However, your method may more robust, and I may switch to it.
> Maybe I am crazy, but the function systemfunc seems to produce
> inconsistent results. It is frustrating me. I am using tcsh as my shell
> - maybe that's the problem, and I should just use bash.
> Matt
>
> Dave Allured wrote:
>> Matt,
>>
>> Here is a method to check for truncated grib files. Use the Unix dd
>> command to get the last four bytes of the file. These bytes should be
>> "7777" in ascii for valid grib files, and something else if the file
>> was truncated.
>>
>> Attached is a stand-alone shell script that you can use. The command
>> line usage is "./eof-check filename".
>>
>> Call this with the NCL systemfunc function, parse the output, and
>> compare with " 7 7 7 7" in an if statement. Due to the output
>> format of the od command, the leading and embedded spaces in this
>> compare string are required.
>>
>> Since grib1 and grib2 formats both use identical end sections, this
>> method should work for both.
>>
>> Caveats:
>>
>> 1. Though extremely unlikely, this method will falsely approve a
>> truncated file that happens to have "7777" in a data array, exactly at
>> the break point. If this becomes a problem, there is a more reliable
>> but complicated method to check grib files.
>>
>> 2. There are variants of the "dd" command out there. You may need to
>> adjust the command arguments for your version.
>>
>> 3. Beware of extra messages from "dd" to standard error. I suspect
>> that they will just print harmlessly to your console. However,
>> standard error handling is not currently defined in the documentation
>> for systemfunc.
>>
>> 4. Some databases have grib files padded with zeros to block
>> boundaries. If you get any of these, you will have to adjust the
>> parsing accordingly.
>>
>> Grib1 reference:
>> http://www.wmo.ch/web/www/WDM/Guides/Guide-binary-2.html
>>
>> Grib2 reference:
>> http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc.shtml
>>
>> I hope this is of some use to you.
>>
>> --Dave A.
>> CU/CIRES Climate Diagnostics Center (CDC)
>> NOAA/ESRL/PSD, Climate Analysis Branch (CAB)
>>
>> Matt Fearon wrote:
>>> Dear NCL Users:
>>>
>>> When I use addfile on a particular grib file, ie,
>>> f=addfile(test.grb,"r"), I receive the following error message:
>>>
>>> fatal:Premature end-of-file, file appears to be truncated
>>>
>>> I realize this is a problem with the file. However, is there an easy
>>> method to check for this error so my script
>>> will not crash. I would like to be able to skip such files. I have
>>> tried the following, but have been
>>> unsuccessful;
>>>
>>> if (any(f.eq."fatal")) then
>>> print("file truncated, skip to next file")
>>> end if
>>>
>>> Any suggestions would be greatly appreciated.
>>> thanks,
>>> Matt
>> ------------------------------------------------------------------------
>>
>> #!/bin/sh
>> #
>> # Show last four bytes of any file.
>> # Used to check for valid terminator '7777' on grib files, among other
>> things.
>> #
>> # 2004-dec-23 Original version by Hoop.
>> # 2007-jun-04 Fix for Mac OS (GNU) version of dd. Change 'iseek'
>> to 'skip'.
>> # 2007-jun-05 Show exactly last four bytes.
>> #
>> FILESIZE=`ls -lL $1 | awk '{print $5}'`
>> SKIPCNT=`expr $FILESIZE - 4`
>> dd if=$1 ibs=1 skip=$SKIPCNT | od -c
>>
>> ------------------------------------------------------------------------
_______________________________________________
ncl-talk mailing list
ncl-talk_at_ucar.edu
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Tue Jun 05 2007 - 15:49:16 MDT