解决loguru在multiprocessing并发场景下子线程日志无法记录到文件问题

dkgee2024-07-06 17:50

logger对象在创建时必须从父进程传递给子进程才能记录日志，有两种传递方式：

**第一种：**通过类定义一个日志变量，然后外部传入logger对象来记录日志。

python 复制代码

# workers_a.py
class Worker:

    _logger = None

    @staticmethod
    def set_logger(logger_):
        Worker._logger = logger_

    def work(self, x):
        self._logger.info("Square rooting {}", x)
        return x**0.5

第二种：通过将logger对象定义为全局变量来记录日志。

python 复制代码

# workers_b.py
from loguru import logger

def set_logger(logger_):
    global logger
    logger = logger_

def work(x):
    logger.info("Square rooting {}", x)
    return x**0.5

在主程序中调用方式如下：

python 复制代码

# main.py
from multiprocessing import Pool
from loguru import logger
import workers_a
import workers_b

if __name__ == "__main__":
    logger.remove()
    logger.add("file.log", enqueue=True)

    worker = workers_a.Worker()
    with Pool(4, initializer=worker.set_logger, initargs=(logger, )) as pool:
        results = pool.map(worker.work, [1, 10, 100])

    with Pool(4, initializer=workers_b.set_logger, initargs=(logger, )) as pool:
        results = pool.map(workers_b.work, [1, 10, 100])

    logger.info("Done")

上面通过multiprocessing 中的Pool创建线程池可记录多线程执行日志。（测试正常执行）上面日志配置时需要把 enqueue=True 放开。

但是上面有个问题：linux系统下的python线程并发与windows环境下的python线程并发调用方式不相同。linux下会通过fork创建，windows是通过spawn拷贝创建。为了避免在windows系统下运行正常的代码在linux系统下出问题，需要借助multiprocessing 的context。代码如下：

python 复制代码

import multiprocessing
from loguru import logger
import workers_a

if __name__ == "__main__":
    context = multiprocessing.get_context("spawn")

    logger.remove()
    logger.add("file.log", enqueue=True, context=context)

    worker = workers_a.Worker()
    with context.Pool(4, initializer=worker.set_logger, initargs=(logger, )) as pool:
        results = pool.map(worker.work, [1, 10, 100])

更详细内容可参考下面地址：Compatibility with multiprocessing using enqueue argument.