Re: Byte array

Hi Dave B.,

Yes Dave, that is how I interpreted it, but the flags then say zero is a
bad value: 0: RayStatus_validity not normal.

I sure can upload a copy of the hdf file, but I sent one to Dennis about a
week ago. Can you check on the FTP to see if there is a hdf file with a
similar file name to: 2008001002019_08922_CS_2C-ICE_GRANULE_P_R04_E02.hdf?
If you guys have cleaned the ftp, I can upload one right away.

/M

On Mon, Jul 29, 2013 at 9:59 PM, David Brown <dbrown@ucar.edu> wrote:

> Hi Marston,
> Judging from the pdf doc that you pointed to, it looks like there should
> no missing value for the Data_quality variable. We could check to see if,
> for some reason,
> NCL is mistakenly adding a _FillValue attribute. Can you send us a sample
> file so we can take a look?
> Thanks.
> -dave
>
>
>
> On Jul 29, 2013, at 1:45 PM, Marston Johnston <shejo284@gmail.com> wrote:
>
> Hi Dave,
>
> Sorry for forgetting to CC the others. I'll do that henceforth.
> Will get back on this issue.
>
> /M
>
>
> On Mon, Jul 29, 2013 at 9:41 PM, Dave Allured - NOAA Affiliate <
> dave.allured@noaa.gov> wrote:
>
>> Hi Marston.
>>
>> This is a good reply, thanks for doing all this investigation. Please
>> remember to CC the user list on all technical replies, unless there is
>> some good reason not to. I will treat this one as private, but would
>> prefer not to keep going this way.
>>
>> I am looking forward to hear what the 2C-ICE people have to say.
>>
>> --Dave
>>
>> On Mon, Jul 29, 2013 at 1:24 PM, Marston Johnston <shejo284@gmail.com>
>> wrote:
>> > Hi Dave A.,
>> >
>> > This is a good point. I'm using NCL 6.1 which cannot convert the HDF4
>> files
>> > to netcdf, therefore I read the hdf files twice, once with the "hdf"
>> file
>> > suffix and then again with "hdf.hdfeos" suffix. This enables NCL to get
>> all
>> > the variables from the HDF4 file. I'm working on a privately-owned linux
>> > cluster so I only have access to the NCL version above. I got a hdf4
>> > converter from the HDF group and converted the hdf4 file to netcdf. It
>> is
>> > this file that I did a netcdf dump on and got '-' for the _FillValue.
>> When I
>> > used CDO to read this netcdf, I get the following:
>> >
>> > 252 : 0000-00-00 00:00:00 0 37082 37082 :
>> > nan : Data_quality
>> >
>> > this reads that all of Data_quality is set to nan.
>> >
>> > I have no software that can read the hdf4 files on the server.
>> >
>> > The Data_Quality doc reads a bit criptic:
>> >
>> > Name in file: Data_quality Source: 2B-GEOPROF 012
>> > Field type (in file): UINT(1) Field type (in algorithm): INT(2)
>> Dimensions:
>> > nray
>> >
>> > Units: --
>> >
>> > Range: 0 to 127 Missing value:
>> > Missing value operator: Factor: 1
>> >
>> > Offset: 0 MB: 0.035
>> >
>> > Flags indicating data quality. If 0, then data is of good quality.
>> > Otherwise, treat as a bit field with 8 flags:
>> >
>> > 0: RayStatus_validity not normal.
>> > 1: GPS data not valid. 2: Temperatures not valid. 3: Radar telemetry
>> data
>> > quality is not normal. 4: Peak power is not normal. 5: CPR calibration
>> > maneuver. 6: Missing frame. 7: Not used.
>> >
>> > You can find this info in the DOC:
>> >
>> http://www.cloudsat.cira.colostate.edu/ICD/2C-ICE/2C-ICE_PDICD.P_R04.20111014.pdf
>> >
>> > I don't have any tools to do view hdfeos files. I'm in contact with the
>> > 2C-ICE people on how to handle the Data_Quality. Waiting to hear back
>> from
>> > them.
>> >
>> >
>> > /M
>> >
>> >
>> > On Mon, Jul 29, 2013 at 8:42 PM, Dave Allured - NOAA Affiliate
>> > <dave.allured@noaa.gov> wrote:
>> >>
>> >> Marston and Dave B,
>> >>
>> >> Marston originally said, "... in the CloudSat docs, the Data_quality
>> >> flag is set to 0 if the data is good." If the _FillValue attribute is
>> >> present and zero after NCL reads this hdf4-eos file, this is totally
>> >> inconsistent with this documentation, and there is an error somewhere.
>> >>
>> >> Possibilities include a special case problem in the NCL driver, the
>> >> hdf4-eos file was modified or produced under different rules, damaged
>> >> file, mistake in the documentation, subtle problem in the user NCL
>> >> script, etc.
>> >>
>> >> I think that Dave B's suggestion to delete the _FillValue attribute in
>> >> NCL memory will mitigate the problem, but I would still wonder how and
>> >> why _FillValue was set to zero. Marston, do you have any software,
>> >> other than NCL, to display the file's metadata and check for the
>> >> expected missing value a different way? I am not experienced with
>> >> HDF4, so I can only raise suspicions, FWIW.
>> >>
>> >> --Dave A.
>> >>
>> >> On Mon, Jul 29, 2013 at 12:00 PM, David Brown <dbrown@ucar.edu> wrote:
>> >> > Hi Marston,
>> >> >
>> >> > You cannot test a value that is set to the _FillValue for equality
>> using
>> >> > e.g.: Data_quality .eq. 0. Instead you should use the 'ismissing'
>> function,
>> >> > for example:
>> >> >
>> >> > where (ismissing(Data_quality), test_var = test_var@_FillValue,
>> >> > test_var)
>> >> >
>> >> > (assumes test_var is another variable in the file that has the same
>> >> > dimensionality as Data_quality)
>> >> >
>> >> > The other option in this case might be to delete the _FillValue
>> >> > attribute:
>> >> >
>> >> > delete (Data_quality@_FillValue)
>> >> > where (Data_quality .eq. 0, test_var = test_var@_FillValue,
>> test_var)
>> >> >
>> >> > This is in no way specific to the byte type.
>> >> >
>> >> > It is a feature of ncdump that values set to the _FillValue are
>> printed
>> >> > as '_'. Again this is not specific to any type. If you don't like
>> this you
>> >> > could use ncl_filedump instead.
>> >> > -dave
>> >> >
>> >> >
>> >> >
>> >> > On Jul 29, 2013, at 11:09 AM, Marston Johnston <shejo284@gmail.com>
>> >> > wrote:
>> >> >
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> I'm working with CloudSat hdf4-eos files using NCL version 6.1.
>> >> >> In the CloudSat file, the Data_Quality variable in the hdf file is
>> of
>> >> >> type "byte".
>> >> >> Is there a function to check which byte is set? For example, how do
>> I
>> >> >> check other variable arrays using mask or where, with the condition
>> that
>> >> >> Data_quality is 0? I tried the function byte2flt, but this fails.
>> The
>> >> >> problem is that in the CloudSat docs, the Data_quality flag is set
>> to 0 if
>> >> >> the data is good. Otherwise the data is bad. After NCL reads the
>> variable,
>> >> >> the _FillValue is set to 0.
>> >> >>
>> >> >> Variable: Data_quality
>> >> >> Type: byte
>> >> >> Total Size: 20820 bytes
>> >> >> 20820 values
>> >> >> Number of Dimensions: 1
>> >> >> Dimensions and sizes: [nray_2C_ICE | 20820]
>> >> >> Coordinates:
>> >> >> Number Of Attributes: 3
>> >> >> _FillValue : 0
>> >> >> unsigned : True
>> >> >> hdfeos_name : Data_quality
>> >> >>
>> >> >> Another strange thing is that when the hdd file is converted to
>> netcdf,
>> >> >> the ncdump shows much of, if not all of the data as containing:
>> '-', not
>> >> >> values.
>> >> >>
>> >> >> Appreciate little info on how to treat byte arrays.
>> >> >>
>> >> >> /M
>> >
>> >
>> >
>> >
>> > --
>> > Only the fruitful thing is true!
>>
>
>
>
> --
> Only the fruitful thing is true!
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
>

-- 
Only the fruitful thing is true!

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Mon Jul 29 14:08:47 2013