Re: open multiple files, but error occured was some HDF5 information.

From: Wei Huang <huangwei_at_nyahnyahspammersnyahnyah>
Date: Wed Jul 10 2013 - 16:32:07 MDT

dyjbean,

It is likely you have hit the soft-limit of file descriptors.

You check this limit:

csh:
limit

ksh:
ulimit -a

You may try to increase this limit with command:

csh:
limit descriptor 1024
(or a bigger number, say 5000)

ksh:
ulimit -S -n 1024

Please let ncl-talk know if this works.

Thanks,

Wei

huangwei@ucar.edu
VETS/CISL
National Center for Atmospheric Research
P.O. Box 3000 (1850 Table Mesa Dr.)
Boulder, CO 80307-3000 USA
(303) 497-8924

On Jul 10, 2013, at 2:03 AM, dyjbean <dyjbean@gmail.com> wrote:

> hi
> i want to open many netcdf files and combine variables in nc files to one file, the following is a part of my code :
>
> load "/public/home/douyj/soft/ncl/lib/ncarg/nclscripts/csm/gsn_code.ncl"
> load "/public/home/douyj/soft/ncl/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
> load "/public/home/douyj/soft/ncl/lib/ncarg/nclscripts/csm/contributed.ncl"
>
> begin
> ;yrs=(/2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012/)
> yrs=(/2001/)
>
> data_indir="/public/home/douyj/result_gldas_noah025subp_3h/gldas_0.25_nc"
> data_outdir="/public/home/douyj/result_gldas_noah025subp_3h/monthly_2001-2012"
>
> do i = 0,dimsizes(yrs)-1
>
> dLWdown = systemfunc("ls "+data_indir+"/"+yrs(i)+"/"+"LWdown"+"/"+"*.nc")
> fLWdown = addfiles(dLWdown,"r")
> ListSetType(fLWdown,"cat")
> vLWdown = fLWdown[:]->FLDS
> ..................
>
> when it execute the line vLWdown above , the following error appeared:
>
>
> fatal:Could not open (/public/home/douyj/result_gldas_noah025subp_3h/gldas_0.25_nc/2001/LWdown/GLDAS_LWdown.200101010600.001.001.nc)
> HDF5-DIAG: Error detected in HDF5 (1.8.8) thread 0:
> #000: H5F.c line 794 in H5Fis_hdf5(): unable to open file
> major: Low-level I/O
> minor: Unable to initialize object
> #001: H5FD.c line 1086 in H5FD_open(): open failed
> major: Virtual File Layer
> minor: Unable to initialize object
> #002: H5FDsec2.c line 362 in H5FD_sec2_open(): unable to open file: name = '/public/home/douyj/result_gldas_noah025subp_3h/gldas_0.25_nc/2001/LWdown/GLDAS_LWdown.200101010300.001.001.nc', errno = 24, error message = 'Too many open files', flags = 0, o_flags = 0
> major: File accessability
> minor: Unable to open file
> HDF5-DIAG: Error detected in HDF5 (1.8.8) thread 0:
> #000: H5F.c line 1522 in H5Fopen(): unable to open file
> major: File accessability
> minor: Unable to open file
> #001: H5F.c line 1211 in H5F_open(): unable to open file: time = Wed Jul 10 08:27:06 2013
> , name = '/public/home/douyj/result_gldas_noah025subp_3h/gldas_0.25_nc/2001/LWdown/GLDAS_LWdown.200101010300.001.001.nc', tent_flags = 0
> major: File accessability
> minor: Unable to open file
> #002: H5FD.c line 1086 in H5FD_open(): open failed
> major: Virtual File Layer
> minor: Unable to initialize object
> #003: H5FDsec2.c line 362 in H5FD_sec2_open(): unable to open file: name = '/public/home/douyj/result_gldas_noah025subp_3h/gldas_0.25_nc/2001/LWdown/GLDAS_LWdown.200101010300.001.001.nc', errno = 24, error message = 'Too many open files', flags = 0, o_flags = 0
> major: File accessability
> minor: Unable to open file
> Unable to open file: </public/home/douyj/result_gldas_noah025subp_3h/gldas_0.25_nc/2001/LWdown/GLDAS_LWdown.200101010300.001.001.nc>, at line: 32
>
>
> all above error info is from HDF5, but what i processed is netcdf.
> i wonder how those problem occured?
>
> thanks
>
>
> dyjbean
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Wed Jul 10 16:32:23 2013

This archive was generated by hypermail 2.1.8 : Fri Jul 12 2013 - 16:37:39 MDT