Re: (no subject)

Gang,

Dealing with files this large requires a solution tailored to your
application. It looks like you have found a limit to what can be
done with addfiles.

You could try making the output interval larger and grouping time
steps, so that NCL does not have to open as many as 1600 files. It
may take some trial and error to find the optimal grouping for use
with NCL.

Would it be easy for you to break up the very large output file by
data variables, rather than by time steps? I suspect that you would
get better results with NCL this way.

What are the types and dimensions of your typical data variables
such as U, V, W, T, TKE, etc.?

Dave Allured
CU/CIRES Climate Diagnostics Center (CDC)
http://cires.colorado.edu/science/centers/cdc/
NOAA/ESRL/PSD, Climate Analysis Branch (CAB)
http://www.cdc.noaa.gov/

Gang Liu wrote:
> Hello,
>
>
>
> My intention is to obtain time series analysis results of some variables
> from the WRF model output during the last half-hour of integral. I set
> the output interval as 1 second, so there are 1800 times of output
> results in total during the integral of half hour. Previously I had them
> written in one file. The NCL was not able to read variables out of the
> mega file close to 300 Gbytes in size encountering memory-related problem.
>
>
>
> Now I have the output results written in 1800 files, i.e. one file for
> one output of 1 second. So every file is less than 0.5 Gbytes in size now.
>
>
>
> I write the NCL code as follows:
>
>
>
> -------------------------------------------------------------------------------------------------------
>
>
>
> begin
>
>
>
> fils=systemfunc("ls wrfout_d01_0001-01-01_03:*")
>
> f=addfiles(fils+".nc","r")
>
> ListSetType(f,"join")
>
>
>
> u0=f[:]->U
>
> v0=f[:]->V
>
> w0=f[:]->W
>
>
>
> th00=f[:]->T
>
> th0=th00+300.0
>
>
>
> tke0=f[:]->TKE
>
> xkmv0=f[:]->XKMV
>
> xkhv0=f[:]->XKHV
>
>
>
> ph0=f[:]->PH
>
> phb0=f[:]->PHB
>
>
>
> ……………………………………………………………
>
>
>
> -------------------------------------------------------------------------------------------------------
>
>
>
> However I got the error message as follows:
>
>
>
> -------------------------------------------------------------------------------------------------------
>
>
>
> Copyright (C) 1995-2007 - All Rights Reserved
>
> University Corporation for Atmospheric Research
>
> NCAR Command Language Version 5.0.0
>
> The use of this software is governed by a License Agreement.
>
> See http://www.ncl.ucar.edu/ for more details.
>
> fatal:Too many open files
>
> fatal:Could not open (wrfout_d01_0001-01-01_03:13:10)
>
> fatal:Too many open files
>
> fatal:Could not open (wrfout_d01_0001-01-01_03:13:09)
>
> fatal:Too many open files
>
> fatal:Could not open (wrfout_d01_0001-01-01_03:13:08)
>
> fatal:Too many open files
>
> fatal:Could not open (wrfout_d01_0001-01-01_03:13:07)
>
> fatal:Too many open files
>
> fatal:Could not open (wrfout_d01_0001-01-01_03:13:06)
>
> ……………………………………………………………
>
> ……………………………………………………………
>
> ……………………………………………………………
>
> fatal:Too many open files
>
> fatal:Could not open (wrfout_d01_0001-01-01_03:00:02)
>
> fatal:Too many open files
>
> fatal:Could not open (wrfout_d01_0001-01-01_03:00:01)
>
>
>
> -------------------------------------------------------------------------------------------------------
>
>
>
> I have uploaded the NCL script file (nclpost.ncl) and one of the data
> files (wrfout_d01_0001-01-01_03:00:01) to the directory ‘incoming’ in
> ftp.cgd.ucar.edu.
>
>
>
> Thanks a lot.
>
>
>
> Gang
>
>
>
> ************************************
> Gang Liu
> Department of Atmospheric Sciences
> Nanjing University
> Nanjing 210093
> P. R. China
> Phone: 86-25-83595185 (o)
> Email: gangliu_at_nju.edu.cn <mailto:gangliu_at_nju.edu.cn>
> ************************************
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Fri Dec 05 2008 - 15:43:31 MST