Re: addfiles - Memory issue?

Hi Gus and Dave,

As of now I have managed to increase the open files limit to 10 times more
(my system can now read around 10K files). Having done that, I would like
my program to be able to process more files since I have to analyze a
larger dataset next week, so I may try the SuppressClose option at that
point. Thanks for your assistance! Both were great suggestions.

Thanks,
Nisha

On Fri, Nov 15, 2013 at 3:14 PM, David Brown <dbrown@ucar.edu> wrote:

> Hi Manisha,
>
> In fact Gus is right that you are running into a system limit for the
> number of open files. You may be able to adjust it as he suggests. However,
> NCL is designed to be able to work within the existing limits by closing
> the files after each access. Unfortunately, this option is not working for
> NetCDF4 files in the current release version of NCL. However, it has been
> recently fixed. The way to access the option is by using the
> "SuppressClose" file option by executing the following line prior to
> calling addfiles:
>
> setfileoption("nc","SuppressClose",False)
>
> If Gus's suggestions don't work for you, you can ask us for a new NCL
> binary with the fix. His warning about memory is important to consider
> though. If you don't have enough, you still will not be able to access the
> complete variable. However, note that the significant memory use does not
> occur until you actually access a variable using, e.g.:
>
> v = f[:]->var_name_in_file
>
> You could open all the files using addfiles but restrict yourself to read
> a subset of the total aggregated variable size at a time in a loop.
> -dave
>
>
> On Nov 15, 2013, at 12:46 PM, Gus Correa <gus@ldeo.columbia.edu> wrote:
>
> Hi Manisha
>
> 30k files * 0.5MB = 15GB.
> Does your computer have this much RAM?
>
> cat /proc/meminfo |grep MemTotal will tell (if you use Linux).
>
> Also, 30,000 files may hit the system limit for open files,
> in case NCL opens them all before closing.
> (I don't know what NCL does.)
>
> 2-3% of 30,000 is about 600-900 files, which sounds on the ballpark of
> a typcal limit for open files (1024 is a typical number).
>
> That 2-3% is also about 300-450MB, but that sounds far too low for any
> current computer, so that may not be where you hit the wall.
>
> limit descriptors [C-shell] or
> ulimit -n [Bash]
> will tell your open files limit.
>
> You can try to increase the open files limit with
> 'limit descriptors 32000' or
> 'ulimit -n 32000'.
> However, that depends also on how far your system administrator
> lets the user limits grow.
>
> If you administer the system yourself, and if it is Linux,
> you can change that limit in /etc/security/limits.conf
> That file has some instructions, and 'man limits.conf' also helps.
> Be careful with the syntax and the numbers there,
> because if you break that file the system may break also.
>
> Everything above failing, you may need to read the data
> in smaller batches, as you said.
>
> I hope this helps,
> Gus Correa
>
> On 11/15/2013 02:04 PM, Manisha Ganeshan wrote:
>
> Hello,
>
> I'm trying to use *addfiles* to read a large dataset which contains
> around 30K netcdf files (each of 0.5 MB size). I get the following
> error, which appears to be related to the intrinsic memory of the NCL
> variable? Is there a solution, other than reading fewer batches of files
> at a time? It looks like I can only read 2-3 % of the data at one time.
> Any help will be appreciated.
>
> Regards,
> Nisha
>
> fatal:Could not open
> (/datal/mganesha/GPS/2006/atmPrf_C001.2006.230.06.55.G01_2010.2640_nc.nc
> <http://atmPrf_C001.2006.230.06.55.G01_2010.2640_nc.nc>)
> HDF5-DIAG: Error detected in HDF5 (1.8.8) thread 0:
> #000: H5F.c line 794 in H5Fis_hdf5(): unable to open file
> major: Low-level I/O
> minor: Unable to initialize object
> #001: H5FD.c line 1086 in H5FD_open(): open failed
> major: Virtual File Layer
> minor: Unable to initialize object
> #002: H5FDsec2.c line 362 in H5FD_sec2_open(): unable to open file:
> name =
> '/datal/mganesha/GPS/2006/atmPrf_C001.2006.230.05.28.G17_2010.2640_nc.nc
> <http://atmPrf_C001.2006.230.05.28.G17_2010.2640_nc.nc>', errno = 24,
> error message = 'Too many open files', flags = 0, o_flags = 0
> major: File accessability
> minor: Unable to open file
> HDF5-DIAG: Error detected in HDF5 (1.8.8) thread 0:
> #000: H5F.c line 1522 in H5Fopen(): unable to open file
> major: File accessability
> minor: Unable to open file
> #001: H5F.c line 1211 in H5F_open(): unable to open file: time = Fri
> Nov 15 18:57:38 2013
> , name =
> '/datal/mganesha/GPS/2006/atmPrf_C001.2006.230.05.28.G17_2010.2640_nc.nc
> <http://atmPrf_C001.2006.230.05.28.G17_2010.2640_nc.nc>', tent_flags = 0
> major: File accessability
> minor: Unable to open file
> #002: H5FD.c line 1086 in H5FD_open(): open failed
> major: Virtual File Layer
> minor: Unable to initialize object
> #003: H5FDsec2.c line 362 in H5FD_sec2_open(): unable to open file:
> name =
> '/datal/mganesha/GPS/2006/atmPrf_C001.2006.230.05.28.G17_2010.2640_nc.nc
> <http://atmPrf_C001.2006.230.05.28.G17_2010.2640_nc.nc>', errno = 24,
> error message = 'Too many open files', flags = 0, o_flags = 0
> major: File accessability
> minor: Unable to open file
> Unable to open file:
> </datal/mganesha/GPS/2006/atmPrf_C001.2006.230.05.28.G17_2010.2640_nc.nc
> <http://atmPrf_C001.2006.230.05.28.G17_2010.2640_nc.nc>>, at line: 32
> fatal:NclHDF5: Unable to open file: <@gG:>, at line: 27304276
>
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
>

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Fri Nov 15 15:35:59 2013