Re: Is there an easy way to read...

Hi Gerald,
     There are a lot of useful examples at the page Dave just sent,
which if pieced together, can accomplish what you want to do:
http://www.ncl.ucar.edu/Applications/read_ascii.shtml

Below is some more information that might be helpful, depending on how
you want to approach this data task. I've attached a script which
fleshes out a few of these aspects.

Since every mixed numeric/alpha ASCII file is different, there's no
'one' silver bullet solution, so it will take some work depending on how
careful you have to treat the data. For instance, you have to decide if
your date/time variables are going to be strings or numeric. Generally,
it's helpful to read the date/time variables as strings, concatenate
them into a combined date/time string like YYYYMMDDHHmmss and then
convert this to a time coordinate like 'seconds since 00:00:00
01-01-1970'. I can e-mail you some conversion routines offline if you're
interested. Or if these are unimportant, you can just convert them to
numeric as they are read.

You also have to decide what your missing values are. NCL has it's own
default _FillValue for each variable, but it's very unlikely that your
ASCII file uses NCL's defaults.

If your file had just one header and the rest of the lines are all
formatted the same way with a common delimiter, then you have used a
neat script that Mary Haley wrote:
        http://www.ncl.ucar.edu/Applications/Scripts/ascii_delim_new.ncl
which is discussed in the 'asc5.txt' example. It's a pretty useful and
fairly automated approach for going from ASCII to netcdf.

In your case, you need to know whether the internal header lines, e.g.,
'LATITUDE: 27.55 . . .', affect the processing of the data below. If
these lines can be ignored, then your task isn't too hard - you would
just strip out those lines and use a read approach like the 'pw.dat'
example, or the approach Dave just sent.

If these internal headers do affect how the subsequent data is to be
stored - I say this because I see that bin counter resetting after each
one - then you would need to take more care to read these as a separate
profiles and increment the profile dimension each time one of those
internal header lines are encountered. To do that, you might want to
allocate each of the variables with dimensions like 'profile' and
'level' (or 'time') and then you'd unpack the variables read 'line by
line' from the file into the appropriate 2d variables which you would
then store in the netcdf. If you don't have to deal with the strings
(looks like they are all missing) or any logical types, it isn't too
hard to write this out to netcdf.

Or if you don't care about having the data stored in 2d arrays in the
netcdf, you could always use the read approach Dave suggests, and just
store the indices. You'd then have to treat the issue of separate
profiles whenever you used the netcdf file though. So it's all a matter
of how much processing you want to do before writing to file, and what
form you want it stored in.

I've attached a script which kind of paints the broad picture of how to
go from the ASCII read to a predefined netcdf file, assuming you wanted
to unpack into 2D arrays before you store it. There's a bit in there
about how to account for strings (which will have to be converted to
character arrays first), but that's rather messy. If the strings aren't
important, it'd be a lot easier to just blast it out to netcdf using the
unstructured method.

Good luck!

Jonathan

On 03/25/2011 01:27 PM, Gerald Creager wrote:
> an ascii file with this format:
>
> #YY MM DD hh mm I Bin Depth Dir Speed ErrVL VerVL %Good3 %Good4
> %GoodE EI1 EI2 EI3 EI4 CM1 CM2 CM3 CM4 Flags
> #yr mo dy hr mn - - m degT cm/s cm/s cm/s % %
> % - - - - - - - - -
> 2010 06 01 00 06 0 0 LATITUDE: 27.55 LONGITUDE: -92.49
> PROFILE_STATUS: 3
> 2010 06 01 00 06 0 1 51.0 249 15.5 -1.0 -2.0 99 99 99
> 208 197 197 203 MM MM MM MM 333393330
> 2010 06 01 00 06 0 2 67.0 270 10.0 -0.5 -1.0 99 99 99
> 185 177 176 184 MM MM MM MM 333393330
> 2010 06 01 00 06 0 3 83.0 204 4.9 2.0 1.0 99 99 99
> 188 191 178 213 MM MM MM MM 333393330
> 2010 06 01 00 06 0 4 99.0 261 9.6 -0.5 -0.5 99 99 99
> 171 213 150 186 MM MM MM MM 333393330
> 2010 06 01 00 06 0 5 115.0 213 8.3 1.5 -0.5 99 99 99
> 161 197 135 199 MM MM MM MM 333393330
> 2010 06 01 00 06 0 6 131.0 249 9.7 -0.5 -1.5 99 99 99
> 147 165 131 183 MM MM MM MM 333393330
> 2010 06 01 00 06 0 7 147.0 279 10.1 0.0 -3.0 99 99 99
> 137 129 128 185 MM MM MM MM 333393330
> 2010 06 01 00 06 0 8 163.0 270 10.5 -1.0 -2.0 99 99 99
> 129 120 124 148 MM MM MM MM 333393330
> 2010 06 01 00 06 0 9 179.0 255 5.7 1.5 -2.0 99 99 99
> 126 116 117 176 MM MM MM MM 333393330
> 2010 06 01 00 06 0 10 195.0 242 8.5 2.0 -1.5 99 99 99
> 124 114 113 181 MM MM MM MM 333393330
> 2010 06 01 00 06 0 11 211.0 259 10.2 2.0 -1.5 99 99 99
> 126 114 115 181 MM MM MM MM 333393330
> 2010 06 01 00 06 0 12 227.0 266 7.0 1.5 -3.0 99 99 99
> 123 115 119 158 MM MM MM MM 333393330
> 2010 06 01 00 06 0 13 243.0 299 5.1 2.5 -3.5 99 99 99
> 116 112 107 144 MM MM MM MM 333393330
> 2010 06 01 00 06 0 14 259.0 305 10.4 0.5 -2.0 99 99 99
> 113 106 101 136 MM MM MM MM 333393330
> 2010 06 01 00 06 0 15 275.0 300 12.1 1.5 -2.0 99 99 99
> 110 99 98 114 MM MM MM MM 333393330
> 2010 06 01 00 06 0 16 291.0 305 10.4 2.0 -1.5 99 99 99
> 108 97 95 122 MM MM MM MM 333393333
> 2010 06 01 00 06 0 17 307.0 319 5.3 1.5 -1.0 99 99
> 99 99 93 93 118 MM MM MM MM 333393333
> 2010 06 01 00 06 0 18 323.0 279 6.6 0.0 -0.5 99 99 99
> 101 88 91 103 MM MM MM MM 333393333
> 2010 06 01 00 06 0 19 339.0 248 5.4 1.0 0.0 99 99 99
> 106 86 96 117 MM MM MM MM 333393333
> 2010 06 01 00 06 0 20 355.0 278 7.6 0.5 0.0 99 99 99
> 104 86 97 109 MM MM MM MM 333393333
> 2010 06 01 00 06 0 21 371.0 295 11.6 0.0 0.5 99 99 99
> 103 88 91 104 MM MM MM MM 333393333
> 2010 06 01 00 06 0 22 387.0 305 12.2 0.0 0.0 99 99
> 99 98 87 87 102 MM MM MM MM 333393333
> 2010 06 01 00 06 0 23 403.0 296 13.9 0.0 0.0 99 99
> 99 98 85 87 96 MM MM MM MM 333393333
> 2010 06 01 00 06 0 24 419.0 282 11.8 0.0 -1.5 99 99 99
> 100 87 87 96 MM MM MM MM 333393333
> 2010 06 01 00 06 0 25 435.0 288 11.1 -0.5 -1.5 99 99 99
> 102 88 90 96 MM MM MM MM 333393333
> 2010 06 01 00 06 0 26 451.0 290 5.9 -0.5 0.0 99 99
> 99 95 85 85 90 MM MM MM MM 333393333
> 2010 06 01 00 06 0 27 467.0 295 7.2 -1.0 -0.5 99 99
> 99 92 83 81 90 MM MM MM MM 333393333
> 2010 06 01 00 06 0 28 483.0 288 9.5 0.0 0.5 99 99
> 99 91 80 81 88 MM MM MM MM 333393333
> 2010 06 01 00 06 0 29 499.0 283 6.7 -1.0 -0.5 99 99
> 99 88 78 77 85 MM MM MM MM 333393333
> 2010 06 01 00 06 0 30 515.0 259 7.6 0.0 0.0 99 99
> 99 85 72 73 79 MM MM MM MM 333393333
> 2010 06 01 00 06 0 31 531.0 270 5.0 -0.5 -0.5 99 99
> 99 79 68 71 73 MM MM MM MM 333393333
> 2010 06 01 00 06 0 32 547.0 278 7.1 0.0 -0.5 99 99
> 99 74 62 63 69 MM MM MM MM 333393333
> 2010 06 01 00 06 0 33 563.0 277 12.1 0.0 -0.5 99 99
> 99 74 57 58 64 MM MM MM MM 333393333
> 2010 06 01 00 06 0 34 579.0 281 12.7 -1.5 -0.5 99 99
> 99 72 56 59 64 MM MM MM MM 333393333
> 2010 06 01 00 06 0 35 595.0 281 10.2 -1.0 -1.5 99 99
> 99 68 54 55 61 MM MM MM MM 333393333
> 2010 06 01 00 06 0 36 611.0 285 7.8 -0.5 -1.0 99 99
> 99 64 50 52 58 MM MM MM MM 333393333
> 2010 06 01 00 06 0 37 627.0 295 7.2 -1.5 -2.0 99 99
> 99 60 48 49 55 MM MM MM MM 333393333
> 2010 06 01 00 06 0 38 643.0 312 7.4 -0.5 -1.0 99 99
> 99 61 48 49 55 MM MM MM MM 333393333
> 2010 06 01 00 06 0 39 659.0 330 6.9 -0.5 -1.5 99 99
> 99 58 48 46 55 MM MM MM MM 333393333
> 2010 06 01 00 06 0 40 675.0 312 6.0 -0.5 -0.5 99 99
> 99 57 46 46 50 MM MM MM MM 333393333
> 2010 06 01 00 06 0 41 691.0 307 7.5 -1.0 -1.0 99 99
> 99 52 41 41 48 MM MM MM MM 333393333
> 2010 06 01 00 06 0 42 707.0 297 4.5 -1.0 -1.5 99 99
> 99 50 37 39 43 MM MM MM MM 333393333
> 2010 06 01 00 06 0 43 723.0 286 3.6 0.0 -1.0 99 99
> 99 46 37 37 43 MM MM MM MM 333393333
> 2010 06 01 00 06 0 44 739.0 326 3.6 -0.5 -1.0 99 99
> 99 46 34 35 40 MM MM MM MM 333393333
> 2010 06 01 00 06 0 45 755.0 349 2.5 0.0 0.0 99 99
> 99 45 33 32 38 MM MM MM MM 333393333
> 2010 06 01 00 06 0 46 771.0 56 1.8 0.0 -1.5 99 99
> 99 42 32 30 41 MM MM MM MM 333393333
> 2010 06 01 00 06 0 47 787.0 40 3.9 -1.0 0.0 99 99
> 99 37 30 26 39 MM MM MM MM 333393333
> 2010 06 01 00 06 0 48 803.0 56 1.8 -1.0 -1.0 99 98
> 99 34 31 23 44 MM MM MM MM 333393333
> 2010 06 01 00 16 0 0 LATITUDE: 27.55 LONGITUDE: -92.49
> PROFILE_STATUS: 3
> 2010 06 01 00 16 0 1 51.0 234 12.9 -1.5 1.0 99 99 99
> 207 198 197 204 MM MM MM MM 333393330
> 2010 06 01 00 16 0 2 67.0 254 8.9 1.5 -1.5 99 99 99
> 189 178 178 184 MM MM MM MM 333393330
> 2010 06 01 00 16 0 3 83.0 210 6.9 2.5 0.5 99 99 99
> 189 192 178 217 MM MM MM MM 333393330
> 2010 06 01 00 16 0 4 99.0 278 7.6 0.5 -1.0 99 99 99
> 168 211 151 189 MM MM MM MM 333393330
> 2010 06 01 00 16 0 5 115.0 228 6.0 3.5 1.5 99 99 99
> 152 196 134 200 MM MM MM MM 333393330
> 2010 06 01 00 16 0 6 131.0 285 17.6 -0.5 -3.0 99 99 99
> 144 167 130 178 MM MM MM MM 333393330
> 2010 06 01 00 16 0 7 147.0 291 11.2 2.0 -2.0 99 99 99
> 137 127 128 179 MM MM MM MM 333393330
> 2010 06 01 00 16 0 8 163.0 263 11.6 -1.0 -1.0 99 99 99
> 130 119 124 144 MM MM MM MM 333393330
>
> and eliminate the extraneous inter-timing position lines, then write
> the sucker out as newcdf?
>
> I know most of this is straightforward. Clipping the extraneous lines
> out is stumping me right now. I AM looking for my O'Reilly book on
> sed/awk, though...
>
> I'm attaching the above segment so it can be seen in it's native form.
> The data are from ADCP platforms, and come one platform to a file.
>
> THanks, gerry
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

-- 
------------------------------------------------------
Jonathan Vigh	
Postdoctoral Fellow, Advanced Study Program
National Center for Atmospheric Research
Mesoscale&  Microscale Meteorology Division
Foothills Lab 3 - Rm. 3081
Office: 303-497-8205
Cell:   720-347-9337
------------------------------------------------------

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk