IOError: [Errno 24] Too many open files:

I have a huge file that I am writing into approximately 450 files. I am getting error as too many files open. I searched the web and found some solution but it is not helping.

import resource
resource.setrlimit(resource.RLIMIT_NOFILE, (1000,-1))
>>> len(pureResponseNames) #Filenames 
434
>>> resource.getrlimit(resource.RLIMIT_NOFILE)
(1000, 9223372036854775807)
>>> output_files = [open(os.path.join(outpathDirTest, fname) + ".txt", "w") for fname in pureResponseNames]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IOError: [Errno 24] Too many open files: 'icd9_737.txt'
>>> 

I also changed ulimit from the command line as below:

$ ulimit -n 1200
$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1200
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 709
virtual memory          (kbytes, -v) unlimited
$ 

I am still getting the same error. PS: I also restarted my system and run the program but with no success.

"Too many open files" errors are always tricky – you not only have to twiddle with ulimit, but you also have to check system-wide limits and OSX-specifics. This SO post gives more information on open files in OSX. (Spoiler alert: the default is 256).

However, it is often easy to limit the number of files that have to be open at the same time. If we look at Stefan Bollman's example, we can easily change that to:

pureResponseNames = ['f'+str(i) for i in range(434)]
outpathDirTest="testCase/"
output_files = [os.path.join(outpathDirTest, fname) + ".txt" for fname in pureResponseNames]

for filename in range(output_files):
    with open(filename, 'w') as f:
        f.write('This is a test of file nr.'+str(i))

You should try $ ulimit -n 50000 instead of 1200.

In case you can't close the file for some reasons(e.g. you're using 3rd party module), you may consider to set based on hard maximum limit instead of predefined hard-coded limit (It will throws ValueError if you try to set hard+1):

import resource
soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
resource.setrlimit(resource.RLIMIT_NOFILE, (hard, hard))

And I want to make it clear that even you manually delete the files created while python process still running, it will still throws such error later.

I changed my ulimit to 4096 from 1024 and it worked. Following is the procedure:

Check your num of descriptors limit using:

ulimit -n

For me it was 1024, and I updated it to 4096 and it worked.

ulimit -n 4096

sudo vim /etc/security/limits.conf

add

*         hard    nofile      500000
*         soft    nofile      500000

to the file.

I strongly discourage you from increasing the ulimit.

1. For example, your database may grow a lot and result in generating many more files than it used to, so much that it would become greater than the limit you fixed and thought was enough.

2. It's a time-consuming/error-prone maintenance task because you would have to make sure that every environment/server has that limit properly set and never changed.

You should ensure that open is used in combination with close or that the with statement is used (which is more pythonic).

Third-party libraries might give you issues (for example, pyPDF2 PdfFileMerger.append keeps files open until the write method is called on it). The way I tracked this is pretty ugly but trying a couple of things on the server while monitoring the number of open files did the trick (my local development computer runs under Mac OS X and server is CentOs):

watch 'lsof | grep "something-created-filenames-have-in-common" | wc -l'

A minimal working example would be nice. I got the same results like ron.rothman using the following script with Python 3.3.2, GCC 4.2.1 on mac 10.6.8. Do you get errors using it?

    import os, sys
    import resource
    resource.setrlimit(resource.RLIMIT_NOFILE, (1000,-1))
    pureResponseNames = ['f'+str(i) for i in range(434)]
    try:
        os.mkdir("testCase")
    except:
        print('Maybe the folder is already there.')
    outpathDirTest="testCase/"
    output_files = [open(os.path.join(outpathDirTest, fname) + ".txt", "w") for fname in pureResponseNames]
    for i in range(len(output_files)):
        output_files[i].write('This is a test of file nr.'+str(i))
        output_files[i].close()