Re: ASCII file with variable header and data

From: Mary Haley <haley_at_nyahnyahspammersnyahnyah>
Date: Tue Oct 15 2013 - 10:25:51 MDT

Brian,

You can try the attached script. Look at the comments for more information. It uses a combination of asciiread and various str_xxx functions to read and parse the data.

I'm not sure how well this script will scale to a million-line file. NCL can be quite expensive when doing string processing.

You didn't say what you wanted to do with the data that is read in. I only printed out the min/max of each column.

I tried the script on a 1,800,000 size file, and it took about 72 seconds.

--Mary

On Oct 15, 2013, at 8:29 AM, Brian McNoldy <bmcnoldy@rsmas.miami.edu> wrote:

> Hello,
> I have large text files that I need to read in, but am having some difficulties (new to the language). I have looked online at the posted examples, but couldn't find any that were as flexible as what I'd like. The file has a header of variable length (lines all start with %), then a line that contains the number of data rows to expect, then the data itself (arranged in 9 space-delimited columns).
> For example:
>
> % header1
> % header2
> % header3
> 5
> 2013 213 729150 2232 -2541 8 34 20 111
> 2013 213 729160 3009 -7643 7 14 20 155
> 2013 213 729160 3335 -7511 7 14 20 225
> 2013 213 729160 2464 -2274 8 53 20 97
> 2013 213 729160 2229 -2536 8 35 20 111
>
> Speed isn't a huge concern, but the number of data lines is on the order of 1 million. The header is in the 30-40 line ballpark. Any guidance would be greatly appreciated!
>
> Thank you,
> Brian
> --
> Brian McNoldy
> Senior Research Associate
> ============================
> University of Miami
> Rosenstiel School of Marine & Atmospheric Science
> Phone: (305) 421-4272
> http://andrew.rsmas.miami.edu/bmcnoldy/
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk

Received on Tue Oct 15 10:26:05 2013

This archive was generated by hypermail 2.1.8 : Tue Oct 22 2013 - 10:35:27 MDT