Re: Byte array

Hi,

The 2C-ICE folks replied that it seems as if NCL doesn't handle properly
the byte array Data_quality when it reads it from the hdf file. Here is his
response:

About the Data Quality question: does NCL overwrite the data quality flags
with zero (the fill value) when converting hdf to netcdf? Even where the
data quality flag is not zero? That would be a problem. When you have the
original, correct values the easiest thing to do in any code that analyzes
CloudSat data is to skip profiles that have a data quality value not equal
to zero. If you want to check the bits then at least skip the profiles
where bits 0, 5, and 6 are equal to one.

I'm not sure how to handle bit shift in NCL to check the individual bits.
My original intention was to throw out all points with a flag value greater
than 0.

/M

On Mon, Jul 29, 2013 at 10:08 PM, Marston Johnston <shejo284@gmail.com>wrote:

> Hi Dave B.,
>
> Yes Dave, that is how I interpreted it, but the flags then say zero is a
> bad value: 0: RayStatus_validity not normal.
>
> I sure can upload a copy of the hdf file, but I sent one to Dennis about a
> week ago. Can you check on the FTP to see if there is a hdf file with a
> similar file name to: 2008001002019_08922_CS_2C-ICE_GRANULE_P_R04_E02.hdf?
> If you guys have cleaned the ftp, I can upload one right away.
>
> /M
>
>
> On Mon, Jul 29, 2013 at 9:59 PM, David Brown <dbrown@ucar.edu> wrote:
>
>> Hi Marston,
>> Judging from the pdf doc that you pointed to, it looks like there should
>> no missing value for the Data_quality variable. We could check to see if,
>> for some reason,
>> NCL is mistakenly adding a _FillValue attribute. Can you send us a sample
>> file so we can take a look?
>> Thanks.
>> -dave
>>
>>
>>
>> On Jul 29, 2013, at 1:45 PM, Marston Johnston <shejo284@gmail.com> wrote:
>>
>> Hi Dave,
>>
>> Sorry for forgetting to CC the others. I'll do that henceforth.
>> Will get back on this issue.
>>
>> /M
>>
>>
>> On Mon, Jul 29, 2013 at 9:41 PM, Dave Allured - NOAA Affiliate <
>> dave.allured@noaa.gov> wrote:
>>
>>> Hi Marston.
>>>
>>> This is a good reply, thanks for doing all this investigation. Please
>>> remember to CC the user list on all technical replies, unless there is
>>> some good reason not to. I will treat this one as private, but would
>>> prefer not to keep going this way.
>>>
>>> I am looking forward to hear what the 2C-ICE people have to say.
>>>
>>> --Dave
>>>
>>> On Mon, Jul 29, 2013 at 1:24 PM, Marston Johnston <shejo284@gmail.com>
>>> wrote:
>>> > Hi Dave A.,
>>> >
>>> > This is a good point. I'm using NCL 6.1 which cannot convert the HDF4
>>> files
>>> > to netcdf, therefore I read the hdf files twice, once with the "hdf"
>>> file
>>> > suffix and then again with "hdf.hdfeos" suffix. This enables NCL to
>>> get all
>>> > the variables from the HDF4 file. I'm working on a privately-owned
>>> linux
>>> > cluster so I only have access to the NCL version above. I got a hdf4
>>> > converter from the HDF group and converted the hdf4 file to netcdf. It
>>> is
>>> > this file that I did a netcdf dump on and got '-' for the _FillValue.
>>> When I
>>> > used CDO to read this netcdf, I get the following:
>>> >
>>> > 252 : 0000-00-00 00:00:00 0 37082 37082 :
>>> > nan : Data_quality
>>> >
>>> > this reads that all of Data_quality is set to nan.
>>> >
>>> > I have no software that can read the hdf4 files on the server.
>>> >
>>> > The Data_Quality doc reads a bit criptic:
>>> >
>>> > Name in file: Data_quality Source: 2B-GEOPROF 012
>>> > Field type (in file): UINT(1) Field type (in algorithm): INT(2)
>>> Dimensions:
>>> > nray
>>> >
>>> > Units: --
>>> >
>>> > Range: 0 to 127 Missing value:
>>> > Missing value operator: Factor: 1
>>> >
>>> > Offset: 0 MB: 0.035
>>> >
>>> > Flags indicating data quality. If 0, then data is of good quality.
>>> > Otherwise, treat as a bit field with 8 flags:
>>> >
>>> > 0: RayStatus_validity not normal.
>>> > 1: GPS data not valid. 2: Temperatures not valid. 3: Radar telemetry
>>> data
>>> > quality is not normal. 4: Peak power is not normal. 5: CPR calibration
>>> > maneuver. 6: Missing frame. 7: Not used.
>>> >
>>> > You can find this info in the DOC:
>>> >
>>> http://www.cloudsat.cira.colostate.edu/ICD/2C-ICE/2C-ICE_PDICD.P_R04.20111014.pdf
>>> >
>>> > I don't have any tools to do view hdfeos files. I'm in contact with the
>>> > 2C-ICE people on how to handle the Data_Quality. Waiting to hear back
>>> from
>>> > them.
>>> >
>>> >
>>> > /M
>>> >
>>> >
>>> > On Mon, Jul 29, 2013 at 8:42 PM, Dave Allured - NOAA Affiliate
>>> > <dave.allured@noaa.gov> wrote:
>>> >>
>>> >> Marston and Dave B,
>>> >>
>>> >> Marston originally said, "... in the CloudSat docs, the Data_quality
>>> >> flag is set to 0 if the data is good." If the _FillValue attribute is
>>> >> present and zero after NCL reads this hdf4-eos file, this is totally
>>> >> inconsistent with this documentation, and there is an error somewhere.
>>> >>
>>> >> Possibilities include a special case problem in the NCL driver, the
>>> >> hdf4-eos file was modified or produced under different rules, damaged
>>> >> file, mistake in the documentation, subtle problem in the user NCL
>>> >> script, etc.
>>> >>
>>> >> I think that Dave B's suggestion to delete the _FillValue attribute in
>>> >> NCL memory will mitigate the problem, but I would still wonder how and
>>> >> why _FillValue was set to zero. Marston, do you have any software,
>>> >> other than NCL, to display the file's metadata and check for the
>>> >> expected missing value a different way? I am not experienced with
>>> >> HDF4, so I can only raise suspicions, FWIW.
>>> >>
>>> >> --Dave A.
>>> >>
>>> >> On Mon, Jul 29, 2013 at 12:00 PM, David Brown <dbrown@ucar.edu>
>>> wrote:
>>> >> > Hi Marston,
>>> >> >
>>> >> > You cannot test a value that is set to the _FillValue for equality
>>> using
>>> >> > e.g.: Data_quality .eq. 0. Instead you should use the 'ismissing'
>>> function,
>>> >> > for example:
>>> >> >
>>> >> > where (ismissing(Data_quality), test_var = test_var@_FillValue,
>>> >> > test_var)
>>> >> >
>>> >> > (assumes test_var is another variable in the file that has the same
>>> >> > dimensionality as Data_quality)
>>> >> >
>>> >> > The other option in this case might be to delete the _FillValue
>>> >> > attribute:
>>> >> >
>>> >> > delete (Data_quality@_FillValue)
>>> >> > where (Data_quality .eq. 0, test_var = test_var@_FillValue,
>>> test_var)
>>> >> >
>>> >> > This is in no way specific to the byte type.
>>> >> >
>>> >> > It is a feature of ncdump that values set to the _FillValue are
>>> printed
>>> >> > as '_'. Again this is not specific to any type. If you don't like
>>> this you
>>> >> > could use ncl_filedump instead.
>>> >> > -dave
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Jul 29, 2013, at 11:09 AM, Marston Johnston <shejo284@gmail.com>
>>> >> > wrote:
>>> >> >
>>> >> >>
>>> >> >> Hi,
>>> >> >>
>>> >> >> I'm working with CloudSat hdf4-eos files using NCL version 6.1.
>>> >> >> In the CloudSat file, the Data_Quality variable in the hdf file is
>>> of
>>> >> >> type "byte".
>>> >> >> Is there a function to check which byte is set? For example, how
>>> do I
>>> >> >> check other variable arrays using mask or where, with the
>>> condition that
>>> >> >> Data_quality is 0? I tried the function byte2flt, but this fails.
>>> The
>>> >> >> problem is that in the CloudSat docs, the Data_quality flag is set
>>> to 0 if
>>> >> >> the data is good. Otherwise the data is bad. After NCL reads the
>>> variable,
>>> >> >> the _FillValue is set to 0.
>>> >> >>
>>> >> >> Variable: Data_quality
>>> >> >> Type: byte
>>> >> >> Total Size: 20820 bytes
>>> >> >> 20820 values
>>> >> >> Number of Dimensions: 1
>>> >> >> Dimensions and sizes: [nray_2C_ICE | 20820]
>>> >> >> Coordinates:
>>> >> >> Number Of Attributes: 3
>>> >> >> _FillValue : 0
>>> >> >> unsigned : True
>>> >> >> hdfeos_name : Data_quality
>>> >> >>
>>> >> >> Another strange thing is that when the hdd file is converted to
>>> netcdf,
>>> >> >> the ncdump shows much of, if not all of the data as containing:
>>> '-', not
>>> >> >> values.
>>> >> >>
>>> >> >> Appreciate little info on how to treat byte arrays.
>>> >> >>
>>> >> >> /M
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Only the fruitful thing is true!
>>>
>>
>>
>>
>> --
>> Only the fruitful thing is true!
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>>
>
>
> --
> Only the fruitful thing is true!
>

-- 
Only the fruitful thing is true!

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Mon Jul 29 16:14:41 2013