Re: (no subject)

Hi Gang,

Dave makes some good points here. Even if you manage to use
'addfiles' successfully using the "SuppressClose" file option that I
suggested in my
last email, there is a fundamental limitation of 2 Gigabytes for the
size of an in-memory variable in NCL. If your variables are of type
float and have 4 dimensions and the time dimension is 1600, that
means that the product of the other 3 dimension sizes can be no more
than 335544 (2147483648 / (1600 * 4 )). The memory error you had
previously may have been the result of trying to going over this
limit. Assuming this is the case, you will have to figure out a way
to process your data in stages.

I agree with Dave that in general it is more efficient to break the
data up by variable rather than by timestep if you have a choice.

FYI, we are planning to remove the 2GB limit for variables where
possible, but it is only possible for 64-bit architecture systems.
  -dave b.

On Dec 5, 2008, at 3:43 PM, Dave Allured wrote:

> Gang,
>
> Dealing with files this large requires a solution tailored to your
> application. It looks like you have found a limit to what can be
> done with addfiles.
>
> You could try making the output interval larger and grouping time
> steps, so that NCL does not have to open as many as 1600 files. It
> may take some trial and error to find the optimal grouping for use
> with NCL.
>
> Would it be easy for you to break up the very large output file by
> data variables, rather than by time steps? I suspect that you
> would get better results with NCL this way.
>
> What are the types and dimensions of your typical data variables
> such as U, V, W, T, TKE, etc.?
>
> Dave Allured
> CU/CIRES Climate Diagnostics Center (CDC)
> http://cires.colorado.edu/science/centers/cdc/
> NOAA/ESRL/PSD, Climate Analysis Branch (CAB)
> http://www.cdc.noaa.gov/
>
> Gang Liu wrote:
>> Hello,
>> My intention is to obtain time series analysis results of some
>> variables from the WRF model output during the last half-hour of
>> integral. I set the output interval as 1 second, so there are 1800
>> times of output results in total during the integral of half hour.
>> Previously I had them written in one file. The NCL was not able to
>> read variables out of the mega file close to 300 Gbytes in size
>> encountering memory-related problem.
>> Now I have the output results written in 1800 files, i.e. one
>> file for one output of 1 second. So every file is less than 0.5
>> Gbytes in size now.
>> I write the NCL code as follows:
>>
>> ---------------------------------------------------------------------
>> ----------------------------------
>> begin
>> fils=systemfunc("ls wrfout_d01_0001-01-01_03:*")
>> f=addfiles(fils+".nc","r")
>> ListSetType(f,"join")
>> u0=f[:]->U
>> v0=f[:]->V
>> w0=f[:]->W
>> th00=f[:]->T
>> th0=th00+300.0
>> tke0=f[:]->TKE
>> xkmv0=f[:]->XKMV
>> xkhv0=f[:]->XKHV
>> ph0=f[:]->PH
>> phb0=f[:]->PHB
>> ……………………………………………………………
>>
>> ---------------------------------------------------------------------
>> ----------------------------------
>> However I got the error message as follows:
>>
>> ---------------------------------------------------------------------
>> ----------------------------------
>> Copyright (C) 1995-2007 - All Rights Reserved
>> University Corporation for Atmospheric Research
>> NCAR Command Language Version 5.0.0
>> The use of this software is governed by a License Agreement.
>> See http://www.ncl.ucar.edu/ for more details.
>> fatal:Too many open files
>> fatal:Could not open (wrfout_d01_0001-01-01_03:13:10)
>> fatal:Too many open files
>> fatal:Could not open (wrfout_d01_0001-01-01_03:13:09)
>> fatal:Too many open files
>> fatal:Could not open (wrfout_d01_0001-01-01_03:13:08)
>> fatal:Too many open files
>> fatal:Could not open (wrfout_d01_0001-01-01_03:13:07)
>> fatal:Too many open files
>> fatal:Could not open (wrfout_d01_0001-01-01_03:13:06)
>> ……………………………………………………………
>> ……………………………………………………………
>> ……………………………………………………………
>> fatal:Too many open files
>> fatal:Could not open (wrfout_d01_0001-01-01_03:00:02)
>> fatal:Too many open files
>> fatal:Could not open (wrfout_d01_0001-01-01_03:00:01)
>>
>> ---------------------------------------------------------------------
>> ----------------------------------
>> I have uploaded the NCL script file (nclpost.ncl) and one of the
>> data files (wrfout_d01_0001-01-01_03:00:01) to the directory
>> ‘incoming’ in ftp.cgd.ucar.edu.
>> Thanks a lot.
>> Gang
>> ************************************
>> Gang Liu
>> Department of Atmospheric Sciences
>> Nanjing University
>> Nanjing 210093
>> P. R. China
>> Phone: 86-25-83595185 (o)
>> Email: gangliu_at_nju.edu.cn <mailto:gangliu_at_nju.edu.cn>
>> ************************************
>> ---------------------------------------------------------------------
>> ---
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Fri Dec 05 2008 - 18:25:34 MST