I am sure one of the NCL core developers will respond
later today.
-- I ran a quick and dirty test on a ** 17.9 GB ** netCDF file. Test script attached. I ran this several times on a multi-user system. %> uname -a Linux tramhill.cgd.ucar.edu 2.6.18-194.11.4.el5 #1 SMP Tue Sep 21 05:04:09 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux The f = addfile("..", "r") was essentially 'instantaneous'. NOTE: *no* data is read by "addfile". This creates a reference (pointer) to the file. The taux=f->TAUX ; (1,2400,3600) was 'instantaneous' The temp=f->TEMP ; (1,42,2400,3600) took 2, 2, 3, 3 seconds Several things could affect your data input: (1) Are you on a multi-user system? When NCL is allocating memory for the input array, are other users also 'competing' for memory? (2) Is the data file on a local file system or on a (say) nfs mounted file system? The later case could affect the input data stream significantly. Sometime ago I ran some tests on an nfs mounted file system. About midnight, there was no timing difference when importing data from a locally mounted file and the nfs mounted file. However, in the middle of the day, the timings were very different. === Ultimately, (almost) all tools (NCL, IDL, Matlab, NCO, CDO,...) that read netCDF are using the standard Unidata software. Cheers! On 10/19/10 4:06 AM, Bjoern Maronga wrote: > Hello, > > I have a problem using the NCL command "addfile" for loading NetCDF files, > that have a size of at about 4 gb (or larger). It takes about 5 minutes to > perform this command (I checked the better performance for smaller datasets). > In contrary, the following commands for loading data into arrays is fast. > > For me it looks like "addfile" actually loads the whole amount of data into > the memory, even though only a small part will be loaded into arrays > afterwards. From my point of view, addfile only needs to read the metadata, > kind of a "ncdump" command, which takes for these files not more time than > for smaller files. I am very surprised by this finding and I'm wondering, if > this makes sense at all. I'm working with NCL for maybe two years now and > never noticed this behavior before. Has anything changed with the addfile > command? Currently, I am using version 5.2.0. > > Best regards, > Björn Maronga >
_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
This archive was generated by hypermail 2.1.8 : Tue Oct 19 2010 - 14:38:00 MDT