文章目录
-
- [从 `print()` 到生产级日志的五个台阶](#从
print()到生产级日志的五个台阶) - [台阶一:`logging.basicConfig()` ------ 统一的输出格式](#台阶一:
logging.basicConfig()—— 统一的输出格式) -
- [`%` 格式化 vs f-string](#
%格式化 vs f-string)
- [`%` 格式化 vs f-string](#
- [台阶二:多 Handler + Formatter ------ 不同输出不同格式](#台阶二:多 Handler + Formatter —— 不同输出不同格式)
- [台阶三:Logger 层级 + Filter ------ 模块化控制](#台阶三:Logger 层级 + Filter —— 模块化控制)
- [台阶四:结构化日志 ------ 让机器读懂日志](#台阶四:结构化日志 —— 让机器读懂日志)
-
- [`python-json-logger`:让标准 `logging` 输出 JSON](#
python-json-logger:让标准logging输出 JSON) - `structlog`:更地道的结构化日志
- 结构化日志的事件命名规范
- [`python-json-logger`:让标准 `logging` 输出 JSON](#
- [台阶五:可观测性集成 ------ 日志、指标、链路追踪三合一](#台阶五:可观测性集成 —— 日志、指标、链路追踪三合一)
-
- [FastAPI + structlog + OpenTelemetry 集成](#FastAPI + structlog + OpenTelemetry 集成)
- 工程实战:日志配置中心化
- 避坑指南
-
- [坑一:在模块级别创建 logger 但配置未初始化](#坑一:在模块级别创建 logger 但配置未初始化)
- 坑二:异常日志丢失堆栈信息
- [坑三:在循环中大量 DEBUG 日志](#坑三:在循环中大量 DEBUG 日志)
- 坑四:日志中的敏感信息泄露
- 日志与可观测性的工程原则
- [从 `print()` 到生产级日志的五个台阶](#从
print()是开发者的第一把锤子------但不是每颗钉子都该用锤子砸。生产环境的日志需要的是"可查询、可聚合、可追溯",而不是一行行看不懂的文本。
从 print() 到生产级日志的五个台阶
几乎所有 Python 程序都从 print() 开始调试:
python
# 阶段零:裸 print
def process_order(order_id):
print(f"Processing order {order_id}")
result = db.query(order_id)
print(f"Got result: {result}")
return result
这个阶段的问题显而易见:无法控制输出目标、无法区分重要性、无法附加结构化信息、无法在投产环境中关闭。
五个台阶逐步升级:
| 台阶 | 工具 | 解决的问题 |
|---|---|---|
| 1 | logging.basicConfig() |
统一输出格式 + 级别控制 |
| 2 | 多 Handler + Formatter | 不同输出目标 + 不同格式 |
| 3 | Logger 层级 + Filter | 模块化日志控制 + 按需过滤 |
| 4 | 结构化日志(JSON) | 机器可读、可聚合查询 |
| 5 | 可观测性集成(OTel) | 日志-指标-链路三合一 |
台阶一:logging.basicConfig() ------ 统一的输出格式
python
import logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
datefmt="%Y-%m-%d %H:%M:%S",
)
logger = logging.getLogger(__name__)
def process_order(order_id: int) -> dict:
logger.info("Processing order %d", order_id)
result = {"id": order_id, "status": "done"}
logger.debug("Order %d details: %r", order_id, result)
return result
process_order(42)
# 输出:2026-05-23 14:30:01 [INFO] __main__: Processing order 42
立即收益:
logger.info("Processing order %d", order_id)使用%格式化------日志模块会延迟格式化,如果日志级别是 WARNING,INFO 消息的字符串拼接不会发生%(name)s自动显示调用模块名logging.DEBUG级别的消息在生产环境默认不输出
% 格式化 vs f-string
python
# ❌ f-string:即使日志被抑制,字符串拼接仍然执行
logger.debug(f"Processing {expensive_computation()}")
# ✅ % 格式化:延迟求值,仅在日志输出时格式化
logger.debug("Processing %s", expensive_computation())
在循环中差异更大------10 万次 logger.debug() + f-string 会浪费大量 CPU 做字符串拼接,而日志级别设置为 INFO 时这些拼接完全被跳过。
台阶二:多 Handler + Formatter ------ 不同输出不同格式
python
import logging
import sys
from logging.handlers import RotatingFileHandler
def setup_logging():
# 创建根 logger
root_logger = logging.getLogger()
root_logger.setLevel(logging.DEBUG)
# Handler 1:控制台------简洁格式,INFO 以上
console_handler = logging.StreamHandler(sys.stdout)
console_handler.setLevel(logging.INFO)
console_handler.setFormatter(logging.Formatter(
"%(asctime)s %(levelname)-8s %(message)s",
datefmt="%H:%M:%S",
))
# Handler 2:文件------完整格式,DEBUG 以上
file_handler = RotatingFileHandler(
"app.log",
maxBytes=10 * 1024 * 1024, # 10 MB
backupCount=5,
encoding="utf-8",
)
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(logging.Formatter(
"%(asctime)s [%(levelname)s] %(name)s:%(lineno)d --- %(message)s",
datefmt="%Y-%m-%d %H:%M:%S",
))
# Handler 3:错误专用文件------ERROR 以上
error_handler = RotatingFileHandler(
"error.log",
maxBytes=5 * 1024 * 1024,
backupCount=3,
)
error_handler.setLevel(logging.ERROR)
error_handler.setFormatter(logging.Formatter(
"%(asctime)s [%(levelname)s] %(name)s:%(lineno)d\n"
" File: %(pathname)s\n"
" Message: %(message)s\n"
" ---"
))
root_logger.addHandler(console_handler)
root_logger.addHandler(file_handler)
root_logger.addHandler(error_handler)
setup_logging()
logger = logging.getLogger("myapp.services.order")
logger.debug("Querying database for order #42") # 仅文件
logger.info("Order #42 processed successfully") # 控制台 + 文件
logger.error("Payment failed for order #42", exc_info=True) # 控制台 + 文件 + 错误文件
Handler 选型指南:
| Handler | 场景 | 配置要点 |
|---|---|---|
StreamHandler |
本地开发、Docker stdout | sys.stdout 而非 sys.stderr(避免与错误混淆) |
FileHandler |
本地文件持久化 | 适合单进程应用 |
RotatingFileHandler |
防止日志撑满磁盘 | 设置 maxBytes + backupCount |
TimedRotatingFileHandler |
按时间切分日志 | when="midnight" 按天轮转 |
SysLogHandler |
Linux 系统日志 | 配合 journald 使用 |
HTTPHandler |
实时日志收集 | 发送到日志聚合服务 |
QueueHandler |
多线程/异步 | 配合 QueueListener 避免 I/O 阻塞 |
台阶三:Logger 层级 + Filter ------ 模块化控制
DEBUG 消息
INFO 消息
WARNING 以上
root logger
level=WARNING
myapp
level=INFO
myapp.services
level=DEBUG
myapp.services.order
level=DEBUG
myapp.services.user
level=INFO
myapp.api
level=INFO
myapp.db
level=WARNING
Handler
→ 控制台 + 文件
Handler
→ 控制台 + 文件
Handler
→ 控制台 + 文件
Logger 名称使用点号分隔形成层级关系。子 logger 的消息会向上传播 到父 logger(除非设置 propagate = False)。
python
import logging
# 根 logger
logging.getLogger().setLevel(logging.WARNING)
# 应用级 logger
app_logger = logging.getLogger("myapp")
app_logger.setLevel(logging.INFO)
# 子模块 logger
services_logger = logging.getLogger("myapp.services")
services_logger.setLevel(logging.DEBUG)
db_logger = logging.getLogger("myapp.db")
db_logger.setLevel(logging.WARNING)
# 使用
logger = logging.getLogger("myapp.services.order")
logger.debug("This will appear") # myapp.services 允许 DEBUG
logger.info("This will appear") # myapp 允许 INFO
logger = logging.getLogger("myapp.db")
logger.debug("This will NOT appear") # myapp.db 限制 WARNING
logger.warning("This will appear")
动态控制日志级别的实战
python
import logging
import os
def configure_logging_from_env():
"""从环境变量读取日志级别配置"""
root_level = os.getenv("LOG_LEVEL", "INFO")
logging.getLogger().setLevel(root_level)
# 允许按模块覆盖
module_overrides = {}
for key, value in os.environ.items():
if key.startswith("LOG_LEVEL_"):
module_name = key[len("LOG_LEVEL_"):].replace("__", ".")
module_overrides[module_name] = value
for module, level in module_overrides.items():
logging.getLogger(module).setLevel(level)
logging.getLogger("myapp").info(
"Overriding log level for %s to %s", module, level
)
# 环境变量示例:
# LOG_LEVEL=WARNING
# LOG_LEVEL_myapp__services=DEBUG → myapp.services 的日志级别设为 DEBUG
# LOG_LEVEL_myapp__db=ERROR → myapp.db 的日志级别设为 ERROR
Filter:比级别更细粒度的控制
Filter 可以在日志级别之外,按内容决定是否输出:
python
import logging
class SensitiveDataFilter(logging.Filter):
"""过滤包含敏感信息的日志"""
SENSITIVE_FIELDS = ["password", "token", "secret", "credit_card"]
def filter(self, record: logging.LogRecord) -> bool:
message = record.getMessage()
for field in self.SENSITIVE_FIELDS:
if field in message.lower():
record.msg = record.msg.replace(field, "***REDACTED***")
return True # 仍然输出,但已脱敏
class RequestIDFilter(logging.Filter):
"""为每条日志注入请求 ID"""
def filter(self, record: logging.LogRecord) -> bool:
import contextvars
request_id = contextvars.ContextVar("request_id", default="N/A").get()
record.request_id = request_id
return True
# 使用
logger = logging.getLogger("myapp.api")
logger.addFilter(SensitiveDataFilter())
logger.addFilter(RequestIDFilter())
# 配置 Formatter 使用 request_id
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter(
"%(asctime)s [%(levelname)s] [%(request_id)s] %(message)s"
))
logger.addHandler(handler)
台阶四:结构化日志 ------ 让机器读懂日志
传统的文本日志:
2026-05-23 14:30:01 [INFO] OrderService: Processing order #42 for user alice@example.com
问题:如果想查询"用户 alice@example.com 的所有订单",只能用 grep,无法高效聚合。
结构化日志用 JSON 输出,每条日志是一个独立的 JSON 对象:
json
{
"timestamp": "2026-05-23T14:30:01.234Z",
"level": "INFO",
"logger": "myapp.services.order",
"message": "Processing order",
"order_id": 42,
"user_email": "alice@example.com",
"request_id": "req-abc123"
}
python-json-logger:让标准 logging 输出 JSON
python
import logging
from pythonjsonlogger import jsonlogger
def setup_json_logging():
handler = logging.StreamHandler()
handler.setFormatter(jsonlogger.JsonFormatter(
"%(asctime)s %(levelname)s %(name)s %(message)s",
timestamp=True,
datefmt="%Y-%m-%dT%H:%M:%S.%fZ",
))
root_logger = logging.getLogger()
root_logger.handlers = [handler]
root_logger.setLevel(logging.INFO)
setup_json_logging()
logger = logging.getLogger("myapp.services.order")
# 使用 extra 传入结构化字段
logger.info(
"Order processed",
extra={
"order_id": 42,
"user_email": "alice@example.com",
"total": 199.99,
"duration_ms": 45,
},
)
# 输出:
# {"asctime": "2026-05-23T14:30:01.234Z", "levelname": "INFO",
# "name": "myapp.services.order", "message": "Order processed",
# "order_id": 42, "user_email": "alice@example.com", "total": 199.99,
# "duration_ms": 45}
structlog:更地道的结构化日志
structlog 提供了比 logging + extra 更自然的 API:
python
import structlog
structlog.configure(
processors=[
structlog.stdlib.filter_by_level,
structlog.stdlib.add_logger_name,
structlog.stdlib.add_log_level,
structlog.stdlib.PositionalArgumentsFormatter(),
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.StackInfoRenderer(),
structlog.processors.format_exc_info,
structlog.processors.UnicodeDecoder(),
structlog.dev.ConsoleRenderer(), # 开发环境:彩色输出
# structlog.processors.JSONRenderer(), # 生产环境:JSON 输出
],
wrapper_class=structlog.stdlib.BoundLogger,
context_class=dict,
logger_factory=structlog.stdlib.LoggerFactory(),
cache_logger_on_first_use=True,
)
logger = structlog.get_logger()
# 绑定上下文------后续日志自动携带
log = logger.bind(request_id="req-abc123", user="alice@example.com")
log.info("order_created", order_id=42, total=199.99)
log.info("payment_processed", transaction_id="tx-789", duration_ms=45)
log.error("payment_failed", transaction_id="tx-790", reason="insufficient_funds")
structlog 的核心优势:
- 绑定上下文(Bound Context) :
logger.bind()创建的 logger 自动携带绑定字段,不需要每条日志都传extra - 处理器管道(Processor Pipeline):日志经过一系列处理器逐步增强,最终由渲染器输出
- 开发/生产双模式 :切换
ConsoleRenderer和JSONRenderer只需要一行配置
结构化日志的事件命名规范
结构化日志的核心是"事件"(event),而不是"消息"(message)。每个事件应该用一个下划线分隔的动词短语命名:
python
# ✅ 好的事件命名
log.info("order_created", order_id=42)
log.info("payment_authorized", amount=199.99)
log.warning("rate_limit_approaching", current_rate=95, limit=100)
log.error("database_connection_failed", host="db01", retry_count=3)
# ❌ 差的事件命名------无法聚合查询
log.info("Order #42 was created successfully by user alice")
log.info("Processing payment of 199.99 for order 42")
事件命名的规范使得在 Elasticsearch / Loki 中可以按 event 字段聚合:
json
// Elasticsearch 查询:过去 1 小时内每种事件的频率
{
"aggs": {
"events": {
"terms": { "field": "event", "size": 20 }
}
},
"query": {
"range": { "timestamp": { "gte": "now-1h" } }
}
}
台阶五:可观测性集成 ------ 日志、指标、链路追踪三合一
日志是"发生了什么",指标是"多少次/多快",链路追踪是"请求经过了哪里"。三者结合才构成完整的可观测性。
HTTP 请求
API Gateway
Order Service
Payment Service
Database
Cache (Redis)
日志
Logs
事件记录
指标
Metrics
计数器/直方图
链路追踪
Traces
Span 树
Loki / ELK
日志聚合
Prometheus
指标监控
Jaeger / Tempo
链路可视化
Grafana
统一仪表盘
FastAPI + structlog + OpenTelemetry 集成
python
"""app/main.py ------ FastAPI 应用,集成了结构化日志和链路追踪"""
import time
import uuid
from contextvars import ContextVar
import structlog
from fastapi import FastAPI, Request
from opentelemetry import trace
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
# ===== 链路追踪初始化 =====
provider = TracerProvider()
provider.add_span_processor(
BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317"))
)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)
# ===== 结构化日志初始化 =====
request_id_ctx: ContextVar[str] = ContextVar("request_id", default="N/A")
def add_trace_ids(_, __, event_dict):
"""为日志注入 OpenTelemetry 的 trace_id 和 span_id"""
span = trace.get_current_span()
if span and span.get_span_context().is_valid:
ctx = span.get_span_context()
event_dict["trace_id"] = format(ctx.trace_id, "032x")
event_dict["span_id"] = format(ctx.span_id, "016x")
event_dict["request_id"] = request_id_ctx.get()
return event_dict
structlog.configure(
processors=[
structlog.stdlib.filter_by_level,
structlog.stdlib.add_logger_name,
structlog.stdlib.add_log_level,
structlog.processors.TimeStamper(fmt="iso"),
add_trace_ids,
structlog.processors.JSONRenderer(),
],
wrapper_class=structlog.stdlib.BoundLogger,
context_class=dict,
logger_factory=structlog.stdlib.LoggerFactory(),
)
logger = structlog.get_logger()
# ===== FastAPI 应用 =====
app = FastAPI()
FastAPIInstrumentor.instrument_app(app)
@app.middleware("http")
async def logging_middleware(request: Request, call_next):
"""为每个请求注入 request_id 并记录日志"""
request_id = request.headers.get("X-Request-ID", str(uuid.uuid4()))
request_id_ctx.set(request_id)
log = logger.bind(
method=request.method,
path=request.url.path,
client_ip=request.client.host if request.client else "unknown",
)
log.info("request_started")
start_time = time.perf_counter()
try:
response = await call_next(request)
duration_ms = (time.perf_counter() - start_time) * 1000
log.info(
"request_completed",
status_code=response.status_code,
duration_ms=round(duration_ms, 2),
)
response.headers["X-Request-ID"] = request_id
return response
except Exception:
duration_ms = (time.perf_counter() - start_time) * 1000
log.exception(
"request_failed",
duration_ms=round(duration_ms, 2),
)
raise
@app.get("/orders/{order_id}")
async def get_order(order_id: int):
log = logger.bind(order_id=order_id)
log.info("order_lookup_started")
# 模拟数据库查询
with tracer.start_as_current_span("db_query") as span:
span.set_attribute("db.statement", f"SELECT * FROM orders WHERE id = {order_id}")
time.sleep(0.05)
log.info("order_lookup_completed")
return {"id": order_id, "status": "shipped", "total": 199.99}
这条中间件的关键行为:
- 从请求头提取
X-Request-ID(或生成新 UUID),注入到日志上下文 - 记录请求开始/完成/失败三种事件,每种附带结构化字段
- 通过 OpenTelemetry 自动注入
trace_id和span_id------用同一个 ID 串联日志、指标、链路追踪
工程实战:日志配置中心化
在一个多模块项目中,每个文件里写 logger = logging.getLogger(__name__) 日志会分散难以管理。集中化管理方案:
单文件配置模块
python
"""myapp/logging_config.py ------ 集中化日志配置"""
import logging.config
import os
from typing import Any
LOG_CONFIG: dict[str, Any] = {
"version": 1,
"disable_existing_loggers": False,
"formatters": {
"json": {
"()": "pythonjsonlogger.jsonlogger.JsonFormatter",
"format": "%(asctime)s %(levelname)s %(name)s %(message)s",
"timestamp": True,
},
"console": {
"format": "%(asctime)s %(levelname)-8s %(name)-30s %(message)s",
"datefmt": "%H:%M:%S",
},
},
"filters": {
"sensitive_data": {
"()": "myapp.filters.SensitiveDataFilter",
},
},
"handlers": {
"console": {
"class": "logging.StreamHandler",
"level": "INFO",
"formatter": "console",
"stream": "ext://sys.stdout",
},
"json_file": {
"class": "logging.handlers.RotatingFileHandler",
"level": "DEBUG",
"formatter": "json",
"filename": "logs/app.jsonl",
"maxBytes": 50 * 1024 * 1024,
"backupCount": 10,
},
"json_stdout": {
"class": "logging.StreamHandler",
"level": "DEBUG",
"formatter": "json",
"stream": "ext://sys.stdout",
},
},
"loggers": {
"myapp": {
"handlers": ["console", "json_file"],
"level": "DEBUG",
"propagate": False,
},
"myapp.services": {
"handlers": ["console", "json_file"],
"level": "DEBUG",
"propagate": False,
},
"myapp.db": {
"handlers": ["console", "json_file"],
"level": "INFO",
"propagate": False,
},
"uvicorn": {
"handlers": ["console"],
"level": "INFO",
"propagate": False,
},
},
"root": {
"handlers": ["console"],
"level": "WARNING",
},
}
def configure():
"""根据环境变量选择日志配置"""
env = os.getenv("APP_ENV", "development")
if env == "production":
# 生产环境:全 JSON 输出到 stdout(Docker 收集)
LOG_CONFIG["root"]["handlers"] = ["json_stdout"]
for logger_cfg in LOG_CONFIG["loggers"].values():
logger_cfg["handlers"] = ["json_stdout"]
logging.config.dictConfig(LOG_CONFIG)
在应用入口调用
python
"""myapp/__init__.py"""
from myapp.logging_config import configure
configure()
# 其他模块只需 getLogger
import logging
logger = logging.getLogger(__name__)
模块中的日志使用
python
"""myapp/services/order.py"""
import logging
import time
from contextlib import contextmanager
logger = logging.getLogger(__name__)
@contextmanager
def log_duration(operation: str, **context):
"""上下文管理器:自动记录操作耗时"""
start = time.perf_counter()
try:
yield
except Exception:
duration = time.perf_counter() - start
logger.exception(
"%s failed after %.2fms",
operation,
duration * 1000,
extra={"operation": operation, "duration_ms": duration * 1000, **context},
)
raise
else:
duration = time.perf_counter() - start
logger.info(
"%s completed in %.2fms",
operation,
duration * 1000,
extra={"operation": operation, "duration_ms": duration * 1000, **context},
)
class OrderService:
def create_order(self, user_id: int, items: list[dict]) -> dict:
with log_duration("order_creation", user_id=user_id, item_count=len(items)):
# 实际创建订单的逻辑
order = {"id": 123, "status": "created"}
logger.info(
"Order created",
extra={"order_id": order["id"], "user_id": user_id},
)
return order
避坑指南
坑一:在模块级别创建 logger 但配置未初始化
python
# ❌ myapp/services/order.py
import logging
logging.basicConfig(level=logging.INFO) # 不------模块不应该配置日志
logger = logging.getLogger(__name__)
# ✅ myapp/services/order.py
import logging
logger = logging.getLogger(__name__) # 只获取 logger,不配置
# 配置在 myapp/__init__.py 入口处统一完成
坑二:异常日志丢失堆栈信息
python
# ❌ 丢失堆栈
try:
risky_operation()
except Exception as e:
logger.error(f"Operation failed: {e}") # 只有错误消息,没有堆栈
# ✅ 保留堆栈
try:
risky_operation()
except Exception:
logger.exception("Operation failed") # 自动附加堆栈
# 等价于
logger.error("Operation failed", exc_info=True)
坑三:在循环中大量 DEBUG 日志
python
# ❌ 即使日志级别是 INFO,f-string 仍然执行
for item in large_list:
logger.debug(f"Processing item {expensive_repr(item)}")
# ✅ 先判断日志级别
if logger.isEnabledFor(logging.DEBUG):
for item in large_list:
logger.debug("Processing item %r", item)
坑四:日志中的敏感信息泄露
python
# ❌ 密码/令牌直接出现在日志中
logger.info("User login: %s", {"username": "alice", "password": "secret123"})
# ✅ 在 Formatter 或 Filter 中脱敏
class PasswordFilter(logging.Filter):
def filter(self, record):
if hasattr(record, "msg") and isinstance(record.msg, str):
import re
record.msg = re.sub(
r'(password["\']?\s*[:=]\s*["\']?)([^\s"\']+)',
r'\1***',
record.msg,
)
return True
日志与可观测性的工程原则
- 三个输出目标:控制台给人看(彩色简洁)、JSON 文件给机器读(结构化完整)、集中平台给运维用(聚合查询)
- 事件命名规范化 :用
snake_case的动词短语(order_created,payment_failed),而非自然语言句子 - 上下文绑定而非逐条传递 :用
structlog.bind()或logging.LoggerAdapter绑定 request_id/user_id,而非每条日志都传extra - 日志级别语义精确 :
DEBUG:开发调试信息,生产环境默认关闭INFO:关键业务流程节点(请求开始/结束、订单创建、支付完成)WARNING:可自动恢复的异常(重试成功、降级、接近限流阈值)ERROR:需要人工介入但系统不崩溃(支付失败、第三方 API 超时)CRITICAL:系统级故障(数据库不可用、内存溢出)
- trace_id 是日志的桥梁 :每条日志携带
trace_id,才能在 Loki 中从错误日志跳转到对应的请求链路 - 日志是最后的手段,指标是第一道防线 :与其写几千行日志后
grep,不如用一个 Prometheus counter 统计错误率
如果这篇文章对构建生产级的日志体系有帮助,点赞收藏让更多人看到!关注专栏,持续获取 Python 进阶干货。