python does not release filehandles to logfile
PythonLoggingPython Problem Overview
I have an application which has to run a number of simulation runs. I want to setup a logging mechanisme where all logrecords are logged in a general.log, and all logs for a simulation run go to run00001.log, .... For this I have defined a class Run. in the __init__()
a new filehandle is added for the runlog.
The problem is that the logfiles for the runs never get released, so after a number of runs the available handles are exhausted and the run crashes.
I've set up some routines to test this as follows
main routine
import Model
try:
myrun = Model.Run('20130315150340_run_49295')
ha = raw_input('enter')
myrun.log.info("some info")
except:
traceback.print_exc(file=sys.stdout)
ha = raw_input('enter3')
The class Run is defined in module Model as follows
import logging
class Run(object):
""" Implements the functionality of a single run. """
def __init__(self, runid):
self.logdir="."
self.runid = runid
self.logFile = os.path.join(self.logdir , self.runid + '.log')
self.log = logging.getLogger('Run'+self.runid)
myformatter = logging.Formatter('%(asctime)s %(name)-12s %(levelname)-8s %(message)s')
myhandler = logging.FileHandler(self.logFile)
myhandler.setLevel(logging.INFO)
myhandler.setFormatter(myformatter)
self.log.addHandler(myhandler)
Then I use the program process explorer to follow the filehandlers. And I see the runlogs appear, but never disappear.
Is there a way I can force this?
Python Solutions
Solution 1 - Python
You need to call .close()
on the filehandler.
When your Run
class completes, call:
handlers = self.log.handlers[:]
for handler in handlers:
self.log.removeHandler(handler)
handler.close()
A file handler will automatically re-open the configured filename every time a new log message arrives, so calling handler.close()
may sometimes appear futile. Removing the handler from the logger stops future log records from being sent to it; in the above code we do this first, to avoid an untimely log message from another thread reopening the handler.
Another answer here suggest you use logging.shutdown()
. However, all that logging.shutdown()
does is call handler.flush()
and handler.close()
, and I'd not recommend using it. It leaves the logging module in a state where you can't use logging.shutdown()
again, not reliably.
Solution 2 - Python
You can also shutdown the logging completely. In that case, file handles are being released:
logging.shutdown()
It closes opened handles of all configured logging handlers.
I needed it to be able to delete a log file after a unit test is finished and I was able to delete it right after the call to the logging.shutdown()
method.
Solution 3 - Python
Probably we'll just want to .close()
the FileHandler
's and not the others, so the accepted answer could be slightly modified like:
for handler in self.log.handlers:
if isinstance(handler, logging.FileHandler):
handler.close()
Also, for the simpler cases where we just have a logger configured with logging.basicConfig()
, the handlers can be retrieved by calling logging.getLogger().handlers
.