Add a custom field to the Python log format string

- Posted by: Barbra Lashbrook - December 14, 2020 Back to CIRA Wiki Home

---

Topic Catagories: Examples

This is from a project that makes use of Python multiprocessing and, as the first comment says, is done to provide a process name field. There are other ways to do this, but this one seemed the best for this application.

# coding: utf-8

# Setup fancy logging to see which process the message is from
# Do this before any other logging.getLogger() calls (?)

# Stock logging modules
import os
import logging

class ProcFilter(logging.Filter):
    procName = 'parent'
    def filter(self, record):
        record.proc_name = self.procName
        return True

# Then use proc_name in your formatting preferences
myformatter = logging.Formatter(
  '%(asctime)s %(levelname)-8s%(proc_name)-9s%(name)s: %(message)s'
)

myhandler = logging.StreamHandler()
myhandler.setFormatter(myformatter)
myhandler.addFilter(ProcFilter())

# The handler will propagate to all loggers when set on the root logger
root_logger = logging.getLogger()
root_logger.setLevel(os.environ.get('LOGLEVEL', 'DEBUG').upper())
root_logger.addHandler(myhandler)

log = logging.getLogger(__name__)

# Stock modules continued
import sys
import multiprocessing
from multiprocessing import Process, JoinableQueue

# Local modules

This can then be used to set proc_name differently for every child when using multiprocessing by passing ProcFilter and a child identifier to the child processes as they are created:

def proc_main(q, ProcFilter, procnum):

    # Customize logging for this child process
    ProcFilter.procName = 'child {:d}'.format(procnum)

    log.info('proc_main started')

Then in the log you get:

2020-12-11 20:55:55,547 INFO    parent   __main__: Starting procs
2020-12-11 20:55:55,554 INFO    child 0  __main__: proc_main started
2020-12-11 20:55:55,557 INFO    child 1  __main__: proc_main started
2020-12-11 20:55:55,559 INFO    child 2  __main__: proc_main started