Re: addfiles - Memory issue?

Hi Manisha

30k files * 0.5MB = 15GB.
Does your computer have this much RAM?

cat /proc/meminfo |grep MemTotal will tell (if you use Linux).

Also, 30,000 files may hit the system limit for open files,
in case NCL opens them all before closing.
(I don't know what NCL does.)

2-3% of 30,000 is about 600-900 files, which sounds on the ballpark of
a typcal limit for open files (1024 is a typical number).

That 2-3% is also about 300-450MB, but that sounds far too low for any
current computer, so that may not be where you hit the wall.

limit descriptors [C-shell] or
ulimit -n [Bash]
will tell your open files limit.

You can try to increase the open files limit with
'limit descriptors 32000' or
'ulimit -n 32000'.
However, that depends also on how far your system administrator
lets the user limits grow.

If you administer the system yourself, and if it is Linux,
you can change that limit in /etc/security/limits.conf
That file has some instructions, and 'man limits.conf' also helps.
Be careful with the syntax and the numbers there,
because if you break that file the system may break also.

Everything above failing, you may need to read the data
in smaller batches, as you said.

I hope this helps,
Gus Correa

On 11/15/2013 02:04 PM, Manisha Ganeshan wrote:
> Hello,
>
> I'm trying to use *addfiles* to read a large dataset which contains
> around 30K netcdf files (each of 0.5 MB size). I get the following
> error, which appears to be related to the intrinsic memory of the NCL
> variable? Is there a solution, other than reading fewer batches of files
> at a time? It looks like I can only read 2-3 % of the data at one time.
> Any help will be appreciated.
>
> Regards,
> Nisha
>
> fatal:Could not open
> (/datal/mganesha/GPS/2006/atmPrf_C001.2006.230.06.55.G01_2010.2640_nc.nc
> <http://atmPrf_C001.2006.230.06.55.G01_2010.2640_nc.nc>)
> HDF5-DIAG: Error detected in HDF5 (1.8.8) thread 0:
> #000: H5F.c line 794 in H5Fis_hdf5(): unable to open file
> major: Low-level I/O
> minor: Unable to initialize object
> #001: H5FD.c line 1086 in H5FD_open(): open failed
> major: Virtual File Layer
> minor: Unable to initialize object
> #002: H5FDsec2.c line 362 in H5FD_sec2_open(): unable to open file:
> name =
> '/datal/mganesha/GPS/2006/atmPrf_C001.2006.230.05.28.G17_2010.2640_nc.nc
> <http://atmPrf_C001.2006.230.05.28.G17_2010.2640_nc.nc>', errno = 24,
> error message = 'Too many open files', flags = 0, o_flags = 0
> major: File accessability
> minor: Unable to open file
> HDF5-DIAG: Error detected in HDF5 (1.8.8) thread 0:
> #000: H5F.c line 1522 in H5Fopen(): unable to open file
> major: File accessability
> minor: Unable to open file
> #001: H5F.c line 1211 in H5F_open(): unable to open file: time = Fri
> Nov 15 18:57:38 2013
> , name =
> '/datal/mganesha/GPS/2006/atmPrf_C001.2006.230.05.28.G17_2010.2640_nc.nc
> <http://atmPrf_C001.2006.230.05.28.G17_2010.2640_nc.nc>', tent_flags = 0
> major: File accessability
> minor: Unable to open file
> #002: H5FD.c line 1086 in H5FD_open(): open failed
> major: Virtual File Layer
> minor: Unable to initialize object
> #003: H5FDsec2.c line 362 in H5FD_sec2_open(): unable to open file:
> name =
> '/datal/mganesha/GPS/2006/atmPrf_C001.2006.230.05.28.G17_2010.2640_nc.nc
> <http://atmPrf_C001.2006.230.05.28.G17_2010.2640_nc.nc>', errno = 24,
> error message = 'Too many open files', flags = 0, o_flags = 0
> major: File accessability
> minor: Unable to open file
> Unable to open file:
> </datal/mganesha/GPS/2006/atmPrf_C001.2006.230.05.28.G17_2010.2640_nc.nc
> <http://atmPrf_C001.2006.230.05.28.G17_2010.2640_nc.nc>>, at line: 32
> fatal:NclHDF5: Unable to open file: <@gG:>, at line: 27304276
>
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Fri Nov 15 12:46:53 2013