Comments on: Python logging from multiple processes

By: andrew cooke

andrew cooke — Mon, 16 Aug 2010 10:05:24 +0000

Thanks for this. Very useful, particularly if you want logs from various machines to be in one place. Also, I knew using twisted was going to be the right way to implement it, so now I have an example to help me learn that…

By: Chugo

Chugo — Fri, 29 Jan 2010 21:29:28 +0000

Hello, I’m trying to run on Windows but I get this trace. Any idea where is the problem?

Traceback (most recent call last):
File \D:Python25Libsite-packages wistedapplicationapp.py\, line 693, in
run

File \D:Python25Libsite-packages wistedscripts wistd.py\, line 23, in ru
nApp

File \D:Python25Libsite-packages wistedapplicationapp.py\, line 411, in
run

File \D:Python25Libsite-packages wistedapplicationapp.py\, line 494, in
createOrGetApplication

— —
File \D:Python25Libsite-packages wistedapplicationapp.py\, line 505, in
getApplication

File \D:Python25Libsite-packages wistedapplicationservice.py\, line 390,
in loadApplication

File \D:Python25Libsite-packages wistedpersistedsob.py\, line 210, in lo
adValueFromFile

File \loggingserver.py\, line 16, in
import loggingprotocol
exceptions.ImportError: No module named loggingprotocol

Failed to load application: No module named loggingprotocol

By: Huy

Huy — Tue, 01 Sep 2009 04:00:59 +0000

Guys,
Thanks for the follow up... I've actually run into conflicting updates from multiple processes before. During those times I was trying to log a few images that were base64 encoded to strings from multiple processes. They were relatively small images, most were icons under 2kb.... However, I did get the occasional 8kb image which seems to be over the limit you've specified? Take a look at the script I wrote below. It basically spawns 5 processes and tries to log 100 messages of a single number to the same log file. Each log message should end with a line break. However, if you scroll through the test-results, you can see that some messages have been truncated because another process is trying to write to the same file at the same time.

Results from code below

log test results
Multiprocess log test


#!/usr/bin/env python
import logging
logging.basicConfig(format='%(message)s', filename='loggingtest.txt')
logging.root.setLevel(logging.INFO)

from multiprocessing import Pool
def f(x):
    for i in range(100):
        logging.info(10000 * str(x) +  "\n")

if __name__ == '__main__':
    p = Pool(5)
    p.map(f, [1,2,3,4,5])

By: Erich Heine

Erich Heine — Sun, 16 Aug 2009 16:11:56 +0000

Why do you need this complicated of a setup? I have never seen multiple processes have problems with loging.FileHandler, as it opens the file in mode=”a” and calls fd.flush() in every emit. In fact, I was recently concerned about this, so I did some tests. My setup was very simple:

running 3 processes, each on a different processor. Each process wrote to the log file in a loop, for some n iterations.

I never had a problem. If can you point me to scenarios where multiple processes actually do cause problems?

By: Vinay Sajip

Vinay Sajip — Sat, 15 Aug 2009 18:19:06 +0000

Although a socket server for logging is not part of the library, one is documented in the Python documentation here:

http://docs.python.org/library/logging.html#sending-and-receiving-logging-events-across-a-network

By: Marius Gedminas

Marius Gedminas — Fri, 14 Aug 2009 14:48:52 +0000

I’d appreciate it if you expanded a bit on what happens if you try to log to the same file from multiple Python processes. Assuming you’re running on a Unix-like OS with the files opened in append mode, log messages smaller than 4K and written in a single syscall, with flushing afterwards, I’d expect things to work just fine.

Also, why sys.argv[1] and not sys.argv[0]?

By: John P. Speno

John P. Speno — Fri, 14 Aug 2009 01:39:19 +0000

Did you just reinvent syslog? O_o

Python’s logging module supports syslog so consider using that if your system supports it.

By: Brandon Corfman

Brandon Corfman — Thu, 13 Aug 2009 20:12:55 +0000

Much appreciated. I have a multiprocessing project that needs this.