Re: Question about addfile command

From: David Brown <dbrown_at_nyahnyahspammersnyahnyah>
Date: Thu Dec 09 2010 - 13:13:04 MST

Hi Gareth,
As Dennis noted offline to you, inputting a GRIB file requires looking at each record in the GRIB file in order to determine its contents. That requires seeking through the file and reading a few hundred bytes of descriptive information at the beginning of each record. Some tools, such as GrADS, have a scheme for creating a sort of catalog of the records in a separate 'gribmap' file using an external tool that does this initial walk through the file. We have long had an intention to do something similar for NCL but other priorities have gotten in the way. However, there is still no getting around the fact that the first time the file is read, each record needs to be accessed and this can take a lot time for a large file.

Concerning your observation that NCL seems to be reading everything in the file rather than just the beginning bytes of each record, that may be influenced by the size of each GRIB record relative to the standard block size which is a property of the system you are running on. In other words, NCL seeks to the beginning of each GRIB record and does a block-sized read. If that block is similar in size to the size of the GRIB record itself then you are correct that NCL will read most of the file. If you send your file per the instructions that Dennis sent, we can take a look to see if anything else is going on in your case. Note that if the file is GRIB2 then the low-level reading is performed by the NCEP grib2c library and we do not have as much control, although we can certainly make suggestions to the library maintainers.

 -dave

On Dec 8, 2010, at 8:47 PM, Gareth Berry wrote:

> Hello!
>
> I have a quick question about the addfile command: I've been reading
> some large (1.7Gb) grib files in a plotting script and it's become
> apparent that the addfile command is taking an outstandingly long time
> to execute. I'm curious as to whether they is a way to speed up the
> process of opening a grib file?
>
> I'm using files from a local server via an nfs mount - from my network
> traffic I noticed that ncl transfers an amount of data about equal in
> size to the file itself. I would of assumed that addfile would not
> transfer this amount of data. I'm using ncl 5.2.1 on MacOSx 5.8, but
> have experienced the same issues with ncl 5.1.1. I don't think the nfs
> mount is the culprit as the long wait is not reduced significantly if
> I run using a file stored on my machine (both take ~1 min to read 1.7
> Gb). Any help or suggestions are most welcome!
>
> Cheers,
>
> Gareth.
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Thu Dec 9 13:13:22 2010

This archive was generated by hypermail 2.1.8 : Fri Dec 10 2010 - 14:58:12 MST