Python logging from multiple processes

Posted on August 13th, 2009

Had a rough day today. I just wanted a log. Not just any log mind you, but one that could handle writes from multiple processes running at the same time. A naive me would have put python’s basic log handler within each process and then watch as each of the processes crash and burn because of conflicting disk write access.

But I’ve learned from my past lessons – now I use the python SocketHandler for my logging needs. Here’s a basic snippet below to get you started.

# ============== Socket Logger =============== #
import logging
import logging.handlers # you NEED this line
logger = logging.getLogger("%s_%s" % (os.getpid(), sys.argv[1]) )
logger.setLevel(logging.DEBUG)
socketHandler = logging.handlers.SocketHandler('localhost',
                    logging.handlers.DEFAULT_TCP_LOGGING_PORT)
logger.addHandler(socketHandler)
# ============== Socket Logger =============== #

In a nutshell, it gets a logger instance, sets the logging level to DEBUG, and then attaches a SocketHandler to itself. This means that whenever any piece of your code calls logger.debug(“this is my log message”) , it will send that message through the pipes to your server sitting on the other end which will handle writing your log messages to disk. On most machines the DEFAULT_TCP_LOGGING_PORT will be 9020 and the server will be sitting on localhost:9020 .

The Log Server

I was surprised that python didn’t already have a canonical implementation of a logging server to interface “SocketHandler” available in the standard library. Luckily there’s a pretty neat project that does just that: python-loggingserver. As a bonus it comes with a web interface to view your logs. On most systems, you can access the website by going to http://localhost:9021/ once you’ve started the server.

It requires the twisted networking library so if you haven’t had that installed do a

sudo easy_install twisted

Then svn checkout the project:

svn checkout http://python-loggingserver.googlecode.com/svn/trunk/ logserver

Start the server:

cd logserver
twistd --pidfile=loggingserver.pid --logfile=logginserver.log --python=loggingserver.py

Now you’re ready to log. You can now use the code snippet posted at the beginning of this article or just use the prepackaged testing script to send messages to the server:

python loggingtest.py this_is_one_process

View the results at http://localhost:9021

Happy logging!

7 responses so far

Brandon Corfman says:

August 13, 2009 at 2:12 pm

Much appreciated. I have a multiprocessing project that needs this.
John P. Speno says:

August 13, 2009 at 7:39 pm

Did you just reinvent syslog? O_o

Python’s logging module supports syslog so consider using that if your system supports it.
Marius Gedminas says:

August 14, 2009 at 8:48 am

I’d appreciate it if you expanded a bit on what happens if you try to log to the same file from multiple Python processes. Assuming you’re running on a Unix-like OS with the files opened in append mode, log messages smaller than 4K and written in a single syscall, with flushing afterwards, I’d expect things to work just fine.

Also, why sys.argv[1] and not sys.argv[0]?
Vinay Sajip says:

August 15, 2009 at 12:19 pm

Although a socket server for logging is not part of the library, one is documented in the Python documentation here:

http://docs.python.org/library/logging.html#sending-and-receiving-logging-events-across-a-network
Erich Heine says:

August 16, 2009 at 10:11 am

Why do you need this complicated of a setup? I have never seen multiple processes have problems with loging.FileHandler, as it opens the file in mode=”a” and calls fd.flush() in every emit. In fact, I was recently concerned about this, so I did some tests. My setup was very simple:

running 3 processes, each on a different processor. Each process wrote to the log file in a loop, for some n iterations.

I never had a problem. If can you point me to scenarios where multiple processes actually do cause problems?
Huy says:

August 31, 2009 at 10:00 pm

Guys,

Thanks for the follow up… I’ve actually run into conflicting updates from multiple processes before. During those times I was trying to log a few images that were base64 encoded to strings from multiple processes. They were relatively small images, most were icons under 2kb…. However, I did get the occasional 8kb image which seems to be over the limit you’ve specified? Take a look at the script I wrote below. It basically spawns 5 processes and tries to log 100 messages of a single number to the same log file. Each log message should end with a line break. However, if you scroll through the test-results, you can see that some messages have been truncated because another process is trying to write to the same file at the same time.

Results from code below

log test results

Multiprocess log test
```
#!/usr/bin/env python
import logging
logging.basicConfig(format='%(message)s', filename='loggingtest.txt')
logging.root.setLevel(logging.INFO)

from multiprocessing import Pool
def f(x):
    for i in range(100):
        logging.info(10000 * str(x) +  "\n")

if __name__ == '__main__':
    p = Pool(5)
    p.map(f, [1,2,3,4,5])
```
Chugo says:

January 29, 2010 at 3:29 pm

Hello, I’m trying to run on Windows but I get this trace. Any idea where is the problem?

Traceback (most recent call last):
File \D:Python25Libsite-packages wistedapplicationapp.py\, line 693, in
run

File \D:Python25Libsite-packages wistedscripts wistd.py\, line 23, in ru
nApp

File \D:Python25Libsite-packages wistedapplicationapp.py\, line 411, in
run

File \D:Python25Libsite-packages wistedapplicationapp.py\, line 494, in
createOrGetApplication

— —
File \D:Python25Libsite-packages wistedapplicationapp.py\, line 505, in
getApplication

File \D:Python25Libsite-packages wistedapplicationservice.py\, line 390,
in loadApplication

File \D:Python25Libsite-packages wistedpersistedsob.py\, line 210, in lo
adValueFromFile

File \loggingserver.py\, line 16, in
import loggingprotocol
exceptions.ImportError: No module named loggingprotocol

Failed to load application: No module named loggingprotocol

Huy Nguyen