Re: the addfile command is very slow

From: Dennis Shea <shea_at_nyahnyahspammersnyahnyah>
Date: Tue Oct 19 2010 - 08:18:40 MDT

I am sure one of the NCL core developers will respond
later today.

--
I ran a quick and dirty test on a ** 17.9 GB  ** netCDF file.
Test script attached. I ran this several times on a multi-user
system.
%> uname -a
Linux tramhill.cgd.ucar.edu 2.6.18-194.11.4.el5 #1 SMP Tue Sep 21 
05:04:09 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
The f = addfile("..", "r") was essentially 'instantaneous'.
NOTE: *no* data is read by "addfile". This creates a
reference (pointer) to the file.
The taux=f->TAUX  ; (1,2400,3600) was 'instantaneous'
The temp=f->TEMP  ; (1,42,2400,3600) took 2, 2, 3, 3 seconds
Several things could affect your data input:
(1)
Are you on a multi-user system? When NCL is allocating memory
for the input array, are other users also 'competing' for memory?
(2)
Is the data file on a local file system or on a (say) nfs
mounted file system? The later case could affect the input
data stream significantly. Sometime ago I ran some tests
on an nfs mounted file system. About midnight, there was
no timing difference when importing data from a locally
mounted file and the nfs mounted file. However, in the
middle of the day, the timings were very different.
===
Ultimately, (almost) all tools (NCL, IDL, Matlab, NCO, CDO,...)
that read netCDF are using the standard Unidata software.
Cheers!
On 10/19/10 4:06 AM, Bjoern Maronga wrote:
> Hello,
>
> I have a problem using the NCL command "addfile" for loading NetCDF files,
> that have a size of at about 4 gb (or larger). It takes about 5 minutes to
> perform this command (I checked the better performance for smaller datasets).
> In contrary, the following commands for loading data into arrays is fast.
>
> For me it looks like "addfile" actually loads the whole amount of data into
> the memory, even though only a small part will be loaded into arrays
> afterwards. From my point of view, addfile only needs to read the metadata,
> kind of a "ncdump" command, which takes for these files not more time than
> for smaller files. I am very surprised by this finding and I'm wondering, if
> this makes sense at all. I'm working with NCL for maybe two years now and
> never noticed this behavior before. Has anything changed with the addfile
> command? Currently, I am using version 5.2.0.
>
> Best regards,
> Björn Maronga
>


_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk

Received on Tue Oct 19 08:18:45 2010

This archive was generated by hypermail 2.1.8 : Tue Oct 19 2010 - 14:38:00 MDT