Re: Byte array

Dear NCL group,

I was looking at this question and I believe I may have some insight.

*Take a look at the description for 2 fields, Data_quality and Data_status:*

Name in file: *Data_quality Range*: 0 to 127
Source: 2B-GEOPROF 012 Missing value:
*Field type (in file): UINT(1) *Missing value operator:
*Field type (in algorithm): INT(2) *Factor: 1
Dimensions: nray Offset: 0
Units: -- MB: 0.035
Flags indicating data quality. If 0, then data is of good quality. *
So, it looks like this field is treated in a dual manner - one way for
0 = "good" data,
anything other value should be interpreted as an (8) bit field

Otherwise, treat as a bit field with 8 flags*:

0: RayStatus_validity not normal.

1: GPS data not valid.

2: Temperatures not valid.

3: Radar telemetry data quality is not normal.

4: Peak power is not normal.

5: CPR calibration maneuver.

6: Missing frame.

7: Not used.

Name in file: *Data_status Range*: 0 to 127
Source: 2B-GEOPROF 012 Missing value:
*Field type (in file): UINT(1)* *Missing value operator*:
*Field type (in algorithm): UINT(1)* Factor: 1
Dimensions: nray Offset: 0
Units: -- MB: 0.035
*This is a bit field that contains data status flags*:

*Bit 0: missing frame (0=false, 1=true)* - This is where you identify
the "fill value" which would be any non-zero value.
Notice here that 0 means that nothing is missing. Other values identify
specific problems with the data.

Bit 1: SOH missing (0=false, 1=true)

Bit 2: GPS data valid (0=false, 1=true)

Bit 3: 1 PPS lost (0=false, 1=true)

Bit 4: Star tracker 1 (0=off, 1=on)

Bit 5: Star tracker 2 (0=off, 1=on)

Bit 6: Coast (0=false, 1=true)

Bit 7: NISC (0=false, 1=true)

Is this information useful?
I work at the NASA Langley ASDC (Atmospheric Science Data Center),
however, I am not a Cloudsat guru.

Mary Jane Saddington

On 7/29/2013 4:08 PM, Marston Johnston wrote:
> Hi Dave B.,
>
> Yes Dave, that is how I interpreted it, but the flags then say zero is
> a bad value: 0: RayStatus_validity not normal.
>
> I sure can upload a copy of the hdf file, but I sent one to Dennis
> about a week ago. Can you check on the FTP to see if there is a hdf
> file with a similar file name
> to: 2008001002019_08922_CS_2C-ICE_GRANULE_P_R04_E02.hdf? If you guys
> have cleaned the ftp, I can upload one right away.
>
> /M
>
>
> On Mon, Jul 29, 2013 at 9:59 PM, David Brown <dbrown@ucar.edu
> <mailto:dbrown@ucar.edu>> wrote:
>
> Hi Marston,
> Judging from the pdf doc that you pointed to, it looks like there
> should no missing value for the Data_quality variable. We could
> check to see if, for some reason,
> NCL is mistakenly adding a _FillValue attribute. Can you send us a
> sample file so we can take a look?
> Thanks.
> -dave
>
>
> On Jul 29, 2013, at 1:45 PM, Marston Johnston <shejo284@gmail.com
> <mailto:shejo284@gmail.com>> wrote:
>
>> Hi Dave,
>>
>> Sorry for forgetting to CC the others. I'll do that henceforth.
>> Will get back on this issue.
>>
>> /M
>>
>>
>> On Mon, Jul 29, 2013 at 9:41 PM, Dave Allured - NOAA Affiliate
>> <dave.allured@noaa.gov <mailto:dave.allured@noaa.gov>> wrote:
>>
>> Hi Marston.
>>
>> This is a good reply, thanks for doing all this
>> investigation. Please
>> remember to CC the user list on all technical replies, unless
>> there is
>> some good reason not to. I will treat this one as private,
>> but would
>> prefer not to keep going this way.
>>
>> I am looking forward to hear what the 2C-ICE people have to say.
>>
>> --Dave
>>
>> On Mon, Jul 29, 2013 at 1:24 PM, Marston Johnston
>> <shejo284@gmail.com <mailto:shejo284@gmail.com>> wrote:
>> > Hi Dave A.,
>> >
>> > This is a good point. I'm using NCL 6.1 which cannot
>> convert the HDF4 files
>> > to netcdf, therefore I read the hdf files twice, once with
>> the "hdf" file
>> > suffix and then again with "hdf.hdfeos" suffix. This
>> enables NCL to get all
>> > the variables from the HDF4 file. I'm working on a
>> privately-owned linux
>> > cluster so I only have access to the NCL version above. I
>> got a hdf4
>> > converter from the HDF group and converted the hdf4 file to
>> netcdf. It is
>> > this file that I did a netcdf dump on and got '-' for the
>> _FillValue. When I
>> > used CDO to read this netcdf, I get the following:
>> >
>> > 252 : 0000-00-00 00:00:00 0 37082 37082 :
>> > nan : Data_quality
>> >
>> > this reads that all of Data_quality is set to nan.
>> >
>> > I have no software that can read the hdf4 files on the server.
>> >
>> > The Data_Quality doc reads a bit criptic:
>> >
>> > Name in file: Data_quality Source: 2B-GEOPROF 012
>> > Field type (in file): UINT(1) Field type (in algorithm):
>> INT(2) Dimensions:
>> > nray
>> >
>> > Units: --
>> >
>> > Range: 0 to 127 Missing value:
>> > Missing value operator: Factor: 1
>> >
>> > Offset: 0 MB: 0.035
>> >
>> > Flags indicating data quality. If 0, then data is of good
>> quality.
>> > Otherwise, treat as a bit field with 8 flags:
>> >
>> > 0: RayStatus_validity not normal.
>> > 1: GPS data not valid. 2: Temperatures not valid. 3: Radar
>> telemetry data
>> > quality is not normal. 4: Peak power is not normal. 5: CPR
>> calibration
>> > maneuver. 6: Missing frame. 7: Not used.
>> >
>> > You can find this info in the DOC:
>> >
>> http://www.cloudsat.cira.colostate.edu/ICD/2C-ICE/2C-ICE_PDICD.P_R04.20111014.pdf
>> >
>> > I don't have any tools to do view hdfeos files. I'm in
>> contact with the
>> > 2C-ICE people on how to handle the Data_Quality. Waiting to
>> hear back from
>> > them.
>> >
>> >
>> > /M
>> >
>> >
>> > On Mon, Jul 29, 2013 at 8:42 PM, Dave Allured - NOAA Affiliate
>> > <dave.allured@noaa.gov <mailto:dave.allured@noaa.gov>> wrote:
>> >>
>> >> Marston and Dave B,
>> >>
>> >> Marston originally said, "... in the CloudSat docs, the
>> Data_quality
>> >> flag is set to 0 if the data is good." If the _FillValue
>> attribute is
>> >> present and zero after NCL reads this hdf4-eos file, this
>> is totally
>> >> inconsistent with this documentation, and there is an
>> error somewhere.
>> >>
>> >> Possibilities include a special case problem in the NCL
>> driver, the
>> >> hdf4-eos file was modified or produced under different
>> rules, damaged
>> >> file, mistake in the documentation, subtle problem in the
>> user NCL
>> >> script, etc.
>> >>
>> >> I think that Dave B's suggestion to delete the _FillValue
>> attribute in
>> >> NCL memory will mitigate the problem, but I would still
>> wonder how and
>> >> why _FillValue was set to zero. Marston, do you have any
>> software,
>> >> other than NCL, to display the file's metadata and check
>> for the
>> >> expected missing value a different way? I am not
>> experienced with
>> >> HDF4, so I can only raise suspicions, FWIW.
>> >>
>> >> --Dave A.
>> >>
>> >> On Mon, Jul 29, 2013 at 12:00 PM, David Brown
>> <dbrown@ucar.edu <mailto:dbrown@ucar.edu>> wrote:
>> >> > Hi Marston,
>> >> >
>> >> > You cannot test a value that is set to the _FillValue
>> for equality using
>> >> > e.g.: Data_quality .eq. 0. Instead you should use the
>> 'ismissing' function,
>> >> > for example:
>> >> >
>> >> > where (ismissing(Data_quality), test_var =
>> test_var@_FillValue,
>> >> > test_var)
>> >> >
>> >> > (assumes test_var is another variable in the file that
>> has the same
>> >> > dimensionality as Data_quality)
>> >> >
>> >> > The other option in this case might be to delete the
>> _FillValue
>> >> > attribute:
>> >> >
>> >> > delete (Data_quality@_FillValue)
>> >> > where (Data_quality .eq. 0, test_var =
>> test_var@_FillValue, test_var)
>> >> >
>> >> > This is in no way specific to the byte type.
>> >> >
>> >> > It is a feature of ncdump that values set to the
>> _FillValue are printed
>> >> > as '_'. Again this is not specific to any type. If you
>> don't like this you
>> >> > could use ncl_filedump instead.
>> >> > -dave
>> >> >
>> >> >
>> >> >
>> >> > On Jul 29, 2013, at 11:09 AM, Marston Johnston
>> <shejo284@gmail.com <mailto:shejo284@gmail.com>>
>> >> > wrote:
>> >> >
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> I'm working with CloudSat hdf4-eos files using NCL
>> version 6.1.
>> >> >> In the CloudSat file, the Data_Quality variable in the
>> hdf file is of
>> >> >> type "byte".
>> >> >> Is there a function to check which byte is set? For
>> example, how do I
>> >> >> check other variable arrays using mask or where, with
>> the condition that
>> >> >> Data_quality is 0? I tried the function byte2flt, but
>> this fails. The
>> >> >> problem is that in the CloudSat docs, the Data_quality
>> flag is set to 0 if
>> >> >> the data is good. Otherwise the data is bad. After NCL
>> reads the variable,
>> >> >> the _FillValue is set to 0.
>> >> >>
>> >> >> Variable: Data_quality
>> >> >> Type: byte
>> >> >> Total Size: 20820 bytes
>> >> >> 20820 values
>> >> >> Number of Dimensions: 1
>> >> >> Dimensions and sizes: [nray_2C_ICE | 20820]
>> >> >> Coordinates:
>> >> >> Number Of Attributes: 3
>> >> >> _FillValue : 0
>> >> >> unsigned : True
>> >> >> hdfeos_name : Data_quality
>> >> >>
>> >> >> Another strange thing is that when the hdd file is
>> converted to netcdf,
>> >> >> the ncdump shows much of, if not all of the data as
>> containing: '-', not
>> >> >> values.
>> >> >>
>> >> >> Appreciate little info on how to treat byte arrays.
>> >> >>
>> >> >> /M
>> >
>> >
>> >
>> >
>> > --
>> > Only the fruitful thing is true!
>>
>>
>>
>>
>> --
>> Only the fruitful thing is true!
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
>
>
> --
> Only the fruitful thing is true!

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Tue Jul 30 08:26:39 2013