Re: large netCDF file writing

From: Dennis Shea <shea_at_nyahnyahspammersnyahnyah>
Date: Wed, 18 Oct 2006 09:27:19 -0600 (MDT)

>I want to write arrays to a large file without waiting until next week.

Of course, as the Unidata people themselves say,
"netCDF software is designed for robustness and flexibility,
 not efficiency"

> Although this may be a crazy thing to do, I have a good reason to try.
> The file size is about 46G, and I am writing 2250x2250 2D arrays and
>2250x2250x100 3D arrays.
>The file has already been created with ncgen. The dimensions and
>variables are thus pre-defined. No attributes are written, e.g. (/*/).
> Although I didn't believe most of these would work, I have tried the
>following with no noticeable difference:
>1. Reversing the order of my variable list, which I am looping through
>as I write each variable.

If the file is "predefined", then the underlying Unidata
software is [I speculate] using 'fseek' to go to a specific
predefined file location. It seems to me the write order
should not matter [much].
>2. Setting the file option "PreFill" to false.
Yes, this should speed the file creation process.
However, has already been done in the "ncgen -x" process?

>3. Copying my output array, which is a subset of an internal NCL
>array, to a temporary variable for write.
>Thoughts, ideas?
>versions and platform:
>NCAR Command Language Version 4.2.0.a033
>netCDF 3.6.0
>uname -m: x86_64

I am sure I am not understanding this correctly. Pls clarify.
My understanding is:

[1] You have used "ncgen" to create a file template based
    on a CDL file. Using "ncgen -x" [ie: no prefill] results
    in much faster file creation.
    [a] Is this done independent of NCL? Eg, Invoking ncgen
        from the command line?
             ncgen -x .....
    [b] You are invoking ncgen from within an NCL script
        via the 'system' command. Hence, components
        of the ascii CDL file used by ncgen are generated within
        an NCL script.
[2] You are the opening the file template created in [1] in NCL
    with the "w" option on addfile
        fout = addfile("", "w")
    I speculate this will take time to open.
    NCL is just invoking the underlying Unidata software.
[3] Now you want to write to the file:

      fout->A(nl,:,:) = (/ a /) ; a(2250,2250)
    or maybe
      fout->B = (/ b /) ; b(100,2250,2250)

Well, that is a start.


ncl-talk mailing list
Received on Wed Oct 18 2006 - 09:27:19 MDT

This archive was generated by hypermail 2.2.0 : Sat Oct 21 2006 - 07:29:03 MDT