Re: passing multiple variables as a set

Hi Rob,

A couple years ago, I had a similar problem to what you describe. I had
to read in and process about 270 variables read from 4 different data
sources with two types of time coordinates, and I had to do this for
about 600 discrete and separate cases (tropical cyclones). I ended up
using the attribute array approach to store the various time-dependent
information as attribute arrays attached to 'container' logical
variables for each data source type. This allowed me to decompose and
compartmentalize my code without having to pass hundreds of variables as
variable arguments back and forth. Then at the higher level of code, I
was able to incrementally process one storm at a time and create
separate netcdf files. Finally, I created code (actually, some of it has
to be dynamically generated because NCL lacks an "eval" statement) to
aggregate all the individual netcdf files into one final data set file.

My application was very specialized, but in the process of doing this, I
did manage to come up with some tools and approaches that handle the
syntatical challengs of writing the contents of these 'containers' out
to netcdf files, where the attribute arrays are written as separate
variables. This was a challenge given that there were hundreds of them,
but I think I came up with some clever approaches. Of course, this
doesn't solve the metadata problem - that would have to be added later.
Dealing with _FillValue's is also important to handle carefully when
using attribute arrays. So this might be more hassle than it's worth for
your application, but I could share some code or ideas if you're interested.

In fact, now that I remember all the challenges and the thousands of
lines of code, it might be better for you to just do it in Matlab or
wait until Fall. :-)

Jonathan

Rob Nicholas wrote:
> Hi all,
>
> Thanks for the various suggestions.
>
> On Tue, Jun 8, 2010 at 12:20 PM, Dave Allured <dave.allured@noaa.gov> wrote:
>
>> Consider using Netcdf files as containers. They do exactly what you
>> describe, except (1) they require space on disk and (2) they are
>> slower than ram. You can even have arrays of Netcdf files via the
>> NCL file handles.
>>
>
> This makes a lot of sense and would definitely work. The major
> drawback I see is that it's syntactically clunky, especially when
> compared with Matlab's approach. I don't think reading and writing
> from disk would pose a serious performance problem for the use I have
> in mind.
>
> On Tue, Jun 8, 2010 at 12:22 PM, Lunde, Bruce N CIV NAVOCEANO, NP1
> <bruce.lunde@navy.mil> wrote:
>
>> Hi Rob, The best I've been able to do is attach
>> items to a variable (either a variable you are actually
>> using or a dummy one such as VAR=True) by means of
>> attributes. You can attach all data types as attributes,
>> as well as 1D and 2D arrays, e.g. VAR@lat2d = lat2d,
>> VAR@name = "dummy", ... You can then pass this around.
>>
>
> This is what I've been doing so far and it works fine but for two
> problems: attributes can't have named dimensions and, if you write the
> variable to a NetCDF file, all of the attributes become 1-D. The
> second part hasn't been much of problem in my situation, but the first
> is a bit of a hassle -- it means creating additional attributes to
> carry around the metadata for the arrays that I'm carrying around as
> attributes!
>
> On Tue, Jun 8, 2010 at 2:04 PM, David Brown <dbrown@ucar.edu> wrote:
>
>> We hope to fully implement the NCL 'list' type soon. This is a
>> container type that can hold data of arbitrary types and shapes,
>> including other lists, allowing for tree-type structures.
>>
>
> The "list" type is *exactly* what I'm looking for. I realize that
> this is an experimental feature
> (http://www.ncl.ucar.edu/Support/talk_archives/2010/1210.html) that
> isn't really usable at this point -- any sense of when it might be
> ready?
>
> ~Rob
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Tue Jun 8 16:12:57 2010