Re: [newbie] vectorized way to write data variable to CSV?

Hi Tom,

Have you looked at write_table?

http://www.ncl.ucar.edu/Document/Functions/Built-in/write_table.shtml

If this doesn't work for you, then you may want to consider writing a C or Fortran function to do the writing, and then use WRAPIT to create a *.so file that you can call from NCL.

See example 5 at:

http://www.ncl.ucar.edu/Document/Tools/WRAPIT.shtml#Example_5

--Mary

On Feb 5, 2013, at 12:08 PM, Tom Roche wrote:

>
> What's the most performant way to write both dimension and data values
> of a datavar to CSV? Why I ask:
>
> I have a large (> 1 GB) netCDF file. I'm only interested in one
> datavar in the file, so I used NCL to "prune" it down to one datavar,
> and write that "pruned output" to a new netCDF file: see
>
> https://bitbucket.org/tlroche/mozart-global-to-aqmeii-na/src/2fb24aedc0e7e7134902704bc38f84f69a81b1e1/prune_IOAPI.ncl?at=master
>
> The pruned datavar is much more tractable: 13 MB, 3.3 Mtuples, see
>
> https://bitbucket.org/tlroche/mozart-global-to-aqmeii-na/downloads/ICON_CB05AE5_US12_2007356_pruned.nc
>
> However I want to further operate on that data in R. While I could
> work directly with the netCDF, I find CSV easier to pull into R ...
> but my current code for doing that (~100 lines starting @ line 219 of
> prune_IOAPI.ncl) is
>
> * very unvectorized: it's a for-loop writing tuples to a 1D vector of
> (text) lines. The tuples are values of the pruned datavar
> sg_datavar_grid(t,l,r,c), and each line has the form
>
> sprinti("%i", r) + "," + \
> sprinti("%i", c) + "," + \
> sprinti("%i", l) + "," + \
> sprinti("%i", t) + "," + \
> sprintf("%f", sg_datavar_grid(t,l,r,c))
>
> * not very performant: on an HPC cluster node (admittedly running in
> home, not as a job) writing the 3.3 Mtuples (66 MB) to a string
> variable
>
> lines = new( (/ 3293784 /), string)
>
> (i.e., no file I/O, just in-memory) requires 19.22 min.
>
> https://bitbucket.org/tlroche/mozart-global-to-aqmeii-na/downloads/ICON_CB05AE5_US12_2007356_pruned.txt
>
> So I suspect my code can be improved. Is there a better way to do this?
>
> TIA, Tom Roche <Tom_Roche@pobox.com>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Thu Feb 7 13:26:26 2013