Re: Byte array

Hi Dave,

Sorry for forgetting to CC the others. I'll do that henceforth.
Will get back on this issue.

/M

On Mon, Jul 29, 2013 at 9:41 PM, Dave Allured - NOAA Affiliate <
dave.allured@noaa.gov> wrote:

> Hi Marston.
>
> This is a good reply, thanks for doing all this investigation. Please
> remember to CC the user list on all technical replies, unless there is
> some good reason not to. I will treat this one as private, but would
> prefer not to keep going this way.
>
> I am looking forward to hear what the 2C-ICE people have to say.
>
> --Dave
>
> On Mon, Jul 29, 2013 at 1:24 PM, Marston Johnston <shejo284@gmail.com>
> wrote:
> > Hi Dave A.,
> >
> > This is a good point. I'm using NCL 6.1 which cannot convert the HDF4
> files
> > to netcdf, therefore I read the hdf files twice, once with the "hdf" file
> > suffix and then again with "hdf.hdfeos" suffix. This enables NCL to get
> all
> > the variables from the HDF4 file. I'm working on a privately-owned linux
> > cluster so I only have access to the NCL version above. I got a hdf4
> > converter from the HDF group and converted the hdf4 file to netcdf. It is
> > this file that I did a netcdf dump on and got '-' for the _FillValue.
> When I
> > used CDO to read this netcdf, I get the following:
> >
> > 252 : 0000-00-00 00:00:00 0 37082 37082 :
> > nan : Data_quality
> >
> > this reads that all of Data_quality is set to nan.
> >
> > I have no software that can read the hdf4 files on the server.
> >
> > The Data_Quality doc reads a bit criptic:
> >
> > Name in file: Data_quality Source: 2B-GEOPROF 012
> > Field type (in file): UINT(1) Field type (in algorithm): INT(2)
> Dimensions:
> > nray
> >
> > Units: --
> >
> > Range: 0 to 127 Missing value:
> > Missing value operator: Factor: 1
> >
> > Offset: 0 MB: 0.035
> >
> > Flags indicating data quality. If 0, then data is of good quality.
> > Otherwise, treat as a bit field with 8 flags:
> >
> > 0: RayStatus_validity not normal.
> > 1: GPS data not valid. 2: Temperatures not valid. 3: Radar telemetry data
> > quality is not normal. 4: Peak power is not normal. 5: CPR calibration
> > maneuver. 6: Missing frame. 7: Not used.
> >
> > You can find this info in the DOC:
> >
> http://www.cloudsat.cira.colostate.edu/ICD/2C-ICE/2C-ICE_PDICD.P_R04.20111014.pdf
> >
> > I don't have any tools to do view hdfeos files. I'm in contact with the
> > 2C-ICE people on how to handle the Data_Quality. Waiting to hear back
> from
> > them.
> >
> >
> > /M
> >
> >
> > On Mon, Jul 29, 2013 at 8:42 PM, Dave Allured - NOAA Affiliate
> > <dave.allured@noaa.gov> wrote:
> >>
> >> Marston and Dave B,
> >>
> >> Marston originally said, "... in the CloudSat docs, the Data_quality
> >> flag is set to 0 if the data is good." If the _FillValue attribute is
> >> present and zero after NCL reads this hdf4-eos file, this is totally
> >> inconsistent with this documentation, and there is an error somewhere.
> >>
> >> Possibilities include a special case problem in the NCL driver, the
> >> hdf4-eos file was modified or produced under different rules, damaged
> >> file, mistake in the documentation, subtle problem in the user NCL
> >> script, etc.
> >>
> >> I think that Dave B's suggestion to delete the _FillValue attribute in
> >> NCL memory will mitigate the problem, but I would still wonder how and
> >> why _FillValue was set to zero. Marston, do you have any software,
> >> other than NCL, to display the file's metadata and check for the
> >> expected missing value a different way? I am not experienced with
> >> HDF4, so I can only raise suspicions, FWIW.
> >>
> >> --Dave A.
> >>
> >> On Mon, Jul 29, 2013 at 12:00 PM, David Brown <dbrown@ucar.edu> wrote:
> >> > Hi Marston,
> >> >
> >> > You cannot test a value that is set to the _FillValue for equality
> using
> >> > e.g.: Data_quality .eq. 0. Instead you should use the 'ismissing'
> function,
> >> > for example:
> >> >
> >> > where (ismissing(Data_quality), test_var = test_var@_FillValue,
> >> > test_var)
> >> >
> >> > (assumes test_var is another variable in the file that has the same
> >> > dimensionality as Data_quality)
> >> >
> >> > The other option in this case might be to delete the _FillValue
> >> > attribute:
> >> >
> >> > delete (Data_quality@_FillValue)
> >> > where (Data_quality .eq. 0, test_var = test_var@_FillValue, test_var)
> >> >
> >> > This is in no way specific to the byte type.
> >> >
> >> > It is a feature of ncdump that values set to the _FillValue are
> printed
> >> > as '_'. Again this is not specific to any type. If you don't like
> this you
> >> > could use ncl_filedump instead.
> >> > -dave
> >> >
> >> >
> >> >
> >> > On Jul 29, 2013, at 11:09 AM, Marston Johnston <shejo284@gmail.com>
> >> > wrote:
> >> >
> >> >>
> >> >> Hi,
> >> >>
> >> >> I'm working with CloudSat hdf4-eos files using NCL version 6.1.
> >> >> In the CloudSat file, the Data_Quality variable in the hdf file is of
> >> >> type "byte".
> >> >> Is there a function to check which byte is set? For example, how do I
> >> >> check other variable arrays using mask or where, with the condition
> that
> >> >> Data_quality is 0? I tried the function byte2flt, but this fails. The
> >> >> problem is that in the CloudSat docs, the Data_quality flag is set
> to 0 if
> >> >> the data is good. Otherwise the data is bad. After NCL reads the
> variable,
> >> >> the _FillValue is set to 0.
> >> >>
> >> >> Variable: Data_quality
> >> >> Type: byte
> >> >> Total Size: 20820 bytes
> >> >> 20820 values
> >> >> Number of Dimensions: 1
> >> >> Dimensions and sizes: [nray_2C_ICE | 20820]
> >> >> Coordinates:
> >> >> Number Of Attributes: 3
> >> >> _FillValue : 0
> >> >> unsigned : True
> >> >> hdfeos_name : Data_quality
> >> >>
> >> >> Another strange thing is that when the hdd file is converted to
> netcdf,
> >> >> the ncdump shows much of, if not all of the data as containing:
> '-', not
> >> >> values.
> >> >>
> >> >> Appreciate little info on how to treat byte arrays.
> >> >>
> >> >> /M
> >
> >
> >
> >
> > --
> > Only the fruitful thing is true!
>

-- 
Only the fruitful thing is true!

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Mon Jul 29 13:45:17 2013