python logging ensure a handler is added only once
PythonLoggingPython Problem Overview
I have a piece of code that is initializing a logger as below.
logger = logging.getLogger()
hdlr = logging.FileHandler('logfile.log')
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
hdlr.setFormatter(formatter)
logger.addHandler(hdlr)
logger.setLevel(logging.DEBUG)
Unfortunately this code is being called multiple times, is there any way I can check to see if the handler already exists - I'd prefer to implement this without having to use a Singleton.
EDIT: Sorry, forgot to mention this is on python 2.5 - cheers, Richard
Python Solutions
Solution 1 - Python
As @offbyone comments, it is possible to add redundant handlers to the same instance of the logger. The python docs for logging say- > "Multiple calls to getLogger() with the same name will return a > reference to the same logger object."
So we don't need to worry about making the implementation a singleton, as it already is.
Unfortunately the same is not true for the handlers associated with the same instance of the logger. There can be duplicate handlers attached.
Example-
-
Copy this code and save it in main.py
import logging print 'inside main.py', print '-'*50 def logger(): print 'initializing logger....' logPath = '.' fileName = 'temp' # configure log formatter logFormatter = logging.Formatter("%(asctime)s [%(filename)s] [%(funcName)s] [%(levelname)s] [%(lineno)d] %(message)s") # configure file handler fileHandler = logging.FileHandler("{0}/{1}.log".format(logPath, fileName)) fileHandler.setFormatter(logFormatter) # configure stream handler consoleHandler = logging.StreamHandler() consoleHandler.setFormatter(logFormatter) # get the logger instance logger = logging.getLogger(__name__) # set the logging level logger.setLevel(logging.DEBUG) print 'adding handlers- ' #if not len(logger.handlers): logger.addHandler(fileHandler) logger.addHandler(consoleHandler) print 'logger initialized....\n' print 'associated handlers - ', len(logger.handlers) for handler in logger.handlers: print handler print return logger main_logger = logger() main_logger.info('utilizing main.py logger.') print 'exiting main.py', print '-'*50 -
and the following code in sub.py
print 'inside sub.py', print '-'*50 print 'importing main.py' import main print 'imported main.py' import logging print 'getting logger instance in sub' sub_logger = main.logger() print 'got logger instance in sub' sub_logger.info("utilizing sub_logger") print 'exiting sub.py', print '-'*50 -
Run sub.py
narayan@y510p:~/code/so$ python sub.py inside sub.py -------------------------------------------------- importing main.py inside main.py -------------------------------------------------- initializing logger.... adding handlers- logger initialized.... associated handlers - 2 <logging.FileHandler object at 0x7f7158740c90> <logging.StreamHandler object at 0x7f7158710b10> 2015-08-04 07:41:01,824 [main.py] [<module>] [INFO] [41] utilizing main.py logger. exiting main.py -------------------------------------------------- imported main.py getting logger instance in sub initializing logger.... adding handlers- logger initialized.... associated handlers - 4 # <===== 4 handlers (duplicates added) <logging.FileHandler object at 0x7f7158740c90> <logging.StreamHandler object at 0x7f7158710b10> <logging.FileHandler object at 0x7f7158710bd0> <logging.StreamHandler object at 0x7f7158710c10> got logger instance in sub 2015-08-04 07:41:01,824 [sub.py] [<module>] [INFO] [10] utilizing sub_logger 2015-08-04 07:41:01,824 [sub.py] [<module>] [INFO] [10] utilizing sub_logger exiting sub.py --------------------------------------------------
Hence multiple calls to the method returning the same logger added duplicate handlers.
Now, for your question-
> is there any way I can check to see if the handler already exists
Yes, there is-
logger.handlers returns a list of all the handlers associated with the given logger.
Before adding handlers to an instance of the logger, make sure not to add duplicate handlers
In main.py, just un-comment the line that says if not len(logger.handlers): and indent the following two lines properly-
if not len(logger.handlers):
logger.addHandler(fileHandler)
logger.addHandler(consoleHandler)
Now again run sub.py
narayan@y510p:~/code/so$ python sub.py
inside sub.py --------------------------------------------------
importing main.py
inside main.py --------------------------------------------------
initializing logger....
adding handlers-
logger initialized....
associated handlers - 2
<logging.FileHandler object at 0x7fd67a891c90>
<logging.StreamHandler object at 0x7fd67a862b10>
2015-08-04 08:14:45,620 [main.py] [<module>] [INFO] [41] utilizing main.py logger.
exiting main.py --------------------------------------------------
imported main.py
getting logger instance in sub
initializing logger....
adding handlers-
logger initialized....
associated handlers - 2 # <===== Still 2 handlers (no duplicates)
<logging.FileHandler object at 0x7fd67a891c90>
<logging.StreamHandler object at 0x7fd67a862b10>
got logger instance in sub
2015-08-04 08:14:45,620 [sub.py] [<module>] [INFO] [10] utilizing sub_logger
exiting sub.py --------------------------------------------------
Further, if you want to limit the type of handlers to be added to the logger instance, you can do something like this-
print 'adding handlers- '
# allows to add only one instance of file handler and stream handler
if len(logger.handlers) > 0:
print 'making sure we do not add duplicate handlers'
for handler in logger.handlers:
# add the handlers to the logger
# makes sure no duplicate handlers are added
if not isinstance(handler, logging.FileHandler) and not isinstance(handler, logging.StreamHandler):
logger.addHandler(fileHandler)
print 'added file handler'
logger.addHandler(consoleHandler)
print 'added stream handler'
else:
logger.addHandler(fileHandler)
logger.addHandler(consoleHandler)
print 'added handlers for the first time'
Hope this helps!
Edit:
> Unfortunately the same is not true for the handlers associated > with the same instance of the logger. There can be duplicate > handlers attached.
It turns out that that above statement is not entirely true.
Let's suppose we have created and configured a logger called 'main_logger' in the main module (which simply configures the logger, doesn't return anything).
# get the logger instance
logger = logging.getLogger("main_logger")
# configuration follows
...
Now in a sub-module, if we create a child logger following the naming hierarchy 'main_logger.sub_module_logger', we don't need to configure it in the sub-module. Just creation of the logger following the naming hierarchy is sufficient.
# get the logger instance
logger = logging.getLogger("main_logger.sub_module_logger")
# no configuration needed
# it inherits the configuration from the parent logger
...
And it won't add duplicate handler as well.
Reference- Using logging in multiple modules
Solution 2 - Python
Well the logger.addHandler() will not add a handler if the handler already exists. To check if the handler is already there you can check the logger.handlers list:
logger = logging.getLogger()
hdlr = logging.FileHandler('logfile.log')
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
hdlr.setFormatter(formatter)
logger.addHandler(hdlr)
logger.setLevel(logging.DEBUG)
print logger.handlers
# [<logging.FileHandler object at 0x14542d0>]
logger.addHandler(hdlr)
print logger.handlers
# [<logging.FileHandler object at 0x14542d0>]
Beside that I will suggest putting this code in your main() function if you have one or in the __init__.py file of your package so to not have to call it each time. I will also suggest that you use a named logger, and do not use the root logger. Something like this:
logger = logging.getLogger(__name__)
...
Hope this was helpful :)
Solution 3 - Python
You can also just check to see whether the handler list is empty. Here's the solution I wound up with:
def setup_logging(self, logfile):
self._logger = logging.getLogger('TestSuite')
self._logger.setLevel(logging.INFO)
host = socket.gethostname().split('.')[0]
if self._logger.handlers == []:
fh = logging.handlers.RotatingFileHandler(logfile,
maxBytes=10*1024*1024,
backupCount=5)
strfmt = "%" + "(asctime)s [%s] " % host + "%" + "(message)s"
fmt = logging.Formatter(strfmt, datefmt="%Y.%m%d %H:%M:%S")
fh.setFormatter(fmt)
self._logger.addHandler(fh)
self._logger.info('-' * (55 - len(host)))
I was seeing the handler added multiple times, so each log message was getting written to the log file more than once, and this fixed it.
Solution 4 - Python
If you are familiar with AWS Lambda, then you might already know that in some contexts, the handlers come pre-configured [1]. Assuming logger.handlers is not empty, is not enough. I recommend setting an attribute on the logger instance, like so:
def init_logger(logger):
if hasattr(logger, 'initialized'):
return logger # No need for addHandler
else:
setattr(logger, 'initialized', True)
# Initialize the logger
# ...
[1] https://stackoverflow.com/questions/37703609/using-python-logging-with-aws-lambda
Solution 5 - Python
Try checking if logger is already set. For example, if this code is inside a function:
logger = None
def init_logger():
global logger
if logger is not None:
#logger has already been initialized
return
logger = logging.getLogger()
hdlr = logging.FileHandler('logfile.log')
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
hdlr.setFormatter(formatter)
logger.addHandler(hdlr)
logger.setLevel(logging.DEBUG)