你的 LLM 服务挂了，能在 5 分钟内找到是哪条 Prompt 触发的、触发了哪个模型、花了多少 Token、用户看到了什么------还是要花 2 小时翻 CloudWatch？
一、为什么 LLM Observability 和普通服务不一样
传统服务的 Observability 已经有了成熟的答案：Prometheus 抓指标、ELK 收日志、Jaeger 做 Trace。但当你把 LLM 塞进来，这套体系就开始出现裂缝。
问题一：延迟分布不正常
普通 HTTP 服务的 P99 可能是 P50 的 2--3 倍。LLM 的 P99 可以是 P50 的 10 倍以上------因为有个叫 TTFT（Time To First Token）的新维度，它和总延迟的相关性只有 0.3 左右。你用传统 percentile 看延迟，什么都看不出来。
问题二：Token 是新的"字节"
CPU 使用率、内存用量、请求数------这些指标在 LLM 场景下都退化成次要指标。真正驱动成本和性能的是 input token、output token、cache hit token。你的计费、限速、成本归因都要建立在 token 层面上，而不是请求层面。
问题三：上下文是有状态的
一个 LLM 请求可能包含：系统 Prompt（50--2000 tokens）+ 历史对话（0--50K tokens）+ 当前用户输入 + 工具返回结果。这些内容如果不在 Trace 里体现，出了问题你就只能猜。但把完整上下文塞进日志，一条日志动辄几十 KB，ELK 扛不住。
问题四：模型本身就是一个黑盒
你可以看到请求进去、响应出来。中间发生了什么------是 temperature 太高导致了幻觉，还是 system prompt 和 user message 产生了歧义------你看不到。所以 LLM Observability 还需要一层叫做"LLM-specific evaluation signals"的东西，这不是 metrics/logs/traces 能覆盖的，但要和它们集成。
本文聚焦解决前三个问题：怎么设计 LLM 场景下的 Metrics、Logs、Traces，让它们能真正告诉你发生了什么。
二、Metrics：你需要哪些指标，怎么暴露
2.1 核心指标集
先建立一个最小可行指标集。以下所有指标都假设你在用 Prometheus + Grafana 体系，如果你用其他监控平台，命名不同但语义相同。
python
# metrics.py - LLM Observability Metrics Registry
from prometheus_client import Counter, Histogram, Gauge, Summary
import time
# --------- 请求层指标 ---------
# 请求总量（按模型、端点、状态分桶）
llm_requests_total = Counter(
'llm_requests_total',
'Total LLM API requests',
['model', 'endpoint', 'status', 'provider']
)
# 完整请求延迟（从发出请求到最后一个 token）
llm_request_duration_seconds = Histogram(
'llm_request_duration_seconds',
'Total request duration including all tokens',
['model', 'endpoint', 'provider'],
buckets=[0.1, 0.25, 0.5, 1.0, 2.0, 5.0, 10.0, 30.0, 60.0, float('inf')]
)
# TTFT - Time To First Token（对流式响应最重要的延迟指标）
llm_time_to_first_token_seconds = Histogram(
'llm_time_to_first_token_seconds',
'Time from request sent to first token received',
['model', 'provider'],
buckets=[0.1, 0.2, 0.5, 1.0, 2.0, 5.0, 10.0, float('inf')]
)
# TPOT - Time Per Output Token（生成速度）
llm_time_per_output_token_ms = Histogram(
'llm_time_per_output_token_ms',
'Average ms per output token',
['model', 'provider'],
buckets=[5, 10, 20, 50, 100, 200, 500, float('inf')]
)
# --------- Token 层指标 ---------
llm_input_tokens_total = Counter(
'llm_input_tokens_total',
'Total input tokens consumed',
['model', 'provider', 'feature'] # feature = 哪个业务功能
)
llm_output_tokens_total = Counter(
'llm_output_tokens_total',
'Total output tokens generated',
['model', 'provider', 'feature']
)
llm_cache_read_tokens_total = Counter(
'llm_cache_read_tokens_total',
'Input tokens served from prompt cache (cheaper)',
['model', 'provider']
)
# Token 效率（output/input 比）- 低于 0.1 可能是 prompt 问题
llm_token_efficiency = Summary(
'llm_token_efficiency',
'Ratio of output tokens to input tokens per request',
['model', 'feature']
)
# --------- 成本层指标 ---------
llm_cost_usd_total = Counter(
'llm_cost_usd_total',
'Estimated cost in USD',
['model', 'provider', 'feature', 'tenant_id']
)
# --------- 错误层指标 ---------
llm_errors_total = Counter(
'llm_errors_total',
'Total LLM errors by type',
['model', 'provider', 'error_type']
# error_type: rate_limit, timeout, context_length, invalid_response,
# provider_error, validation_error
)
llm_retries_total = Counter(
'llm_retries_total',
'Total retry attempts',
['model', 'provider', 'reason']
)
# --------- 并发/队列层指标 ---------
llm_requests_in_flight = Gauge(
'llm_requests_in_flight',
'Currently active LLM requests',
['model', 'provider']
)
llm_queue_depth = Gauge(
'llm_queue_depth',
'Requests waiting in queue',
['model', 'priority']
)2.2 指标记录的正确姿势
很多团队做指标的时候犯同一个错误：把指标记录散落在各处，导致 labels 不一致。正确做法是用一个统一的 LLMMetricsRecorder：
python
# metrics_recorder.py
import time
import asyncio
from contextlib import asynccontextmanager
from dataclasses import dataclass, field
from typing import Optional
from metrics import * # 上面定义的所有指标
@dataclass
class LLMRequestContext:
model: str
provider: str
endpoint: str
feature: str
tenant_id: str = "default"
trace_id: Optional[str] = None
@dataclass
class LLMRequestResult:
status: str # "success" | "error"
input_tokens: int = 0
output_tokens: int = 0
cache_read_tokens: int = 0
error_type: Optional[str] = None
first_token_at: Optional[float] = None # unix timestamp
output_token_count: int = 0
# 成本计算（示例：DeepSeek-V3 定价参考）
PRICE_PER_1K_INPUT = 0.001
PRICE_PER_1K_OUTPUT = 0.002
PRICE_PER_1K_CACHE_READ = 0.0001
def estimated_cost_usd(self, model: str) -> float:
# 真实实现应从 price table 查询
return (
self.input_tokens / 1000 * self.PRICE_PER_1K_INPUT +
self.output_tokens / 1000 * self.PRICE_PER_1K_OUTPUT +
self.cache_read_tokens / 1000 * self.PRICE_PER_1K_CACHE_READ
)
class LLMMetricsRecorder:
@asynccontextmanager
async def record_request(self, ctx: LLMRequestContext):
start = time.monotonic()
llm_requests_in_flight.labels(
model=ctx.model, provider=ctx.provider
).inc()
result = LLMRequestResult(status="success")
try:
yield result # 调用方在这里填充 result
except Exception as e:
result.status = "error"
if not result.error_type:
result.error_type = type(e).__name__
raise
finally:
duration = time.monotonic() - start
# 请求计数
llm_requests_total.labels(
model=ctx.model,
endpoint=ctx.endpoint,
status=result.status,
provider=ctx.provider
).inc()
# 延迟
llm_request_duration_seconds.labels(
model=ctx.model,
endpoint=ctx.endpoint,
provider=ctx.provider
).observe(duration)
# TTFT（如果是流式）
if result.first_token_at:
ttft = result.first_token_at - start
llm_time_to_first_token_seconds.labels(
model=ctx.model, provider=ctx.provider
).observe(ttft)
# TPOT
if result.output_token_count > 1:
tpot_ms = (duration - ttft) / (result.output_token_count - 1) * 1000
llm_time_per_output_token_ms.labels(
model=ctx.model, provider=ctx.provider
).observe(tpot_ms)
# Token 计数
if result.input_tokens:
llm_input_tokens_total.labels(
model=ctx.model, provider=ctx.provider, feature=ctx.feature
).inc(result.input_tokens)
if result.output_tokens:
llm_output_tokens_total.labels(
model=ctx.model, provider=ctx.provider, feature=ctx.feature
).inc(result.output_tokens)
if result.cache_read_tokens:
llm_cache_read_tokens_total.labels(
model=ctx.model, provider=ctx.provider
).inc(result.cache_read_tokens)
# Token 效率
if result.input_tokens > 0:
efficiency = result.output_tokens / result.input_tokens
llm_token_efficiency.labels(
model=ctx.model, feature=ctx.feature
).observe(efficiency)
# 成本
cost = result.estimated_cost_usd(ctx.model)
llm_cost_usd_total.labels(
model=ctx.model,
provider=ctx.provider,
feature=ctx.feature,
tenant_id=ctx.tenant_id
).inc(cost)
# 错误
if result.status == "error" and result.error_type:
llm_errors_total.labels(
model=ctx.model,
provider=ctx.provider,
error_type=result.error_type
).inc()
# 释放 in-flight
llm_requests_in_flight.labels(
model=ctx.model, provider=ctx.provider
).dec()
# 使用示例
recorder = LLMMetricsRecorder()
async def call_llm_with_metrics(prompt: str, tenant_id: str):
ctx = LLMRequestContext(
model="deepseek-v3",
provider="deepseek",
endpoint="/v1/chat/completions",
feature="code_review",
tenant_id=tenant_id,
trace_id=get_current_trace_id()
)
async with recorder.record_request(ctx) as result:
response = await deepseek_client.chat.completions.create(
model=ctx.model,
messages=[{"role": "user", "content": prompt}],
stream=True
)
first_token = False
async for chunk in response:
if chunk.choices and not first_token:
result.first_token_at = time.monotonic()
first_token = True
if chunk.choices:
result.output_token_count += 1
result.input_tokens = response.usage.prompt_tokens
result.output_tokens = response.usage.completion_tokens
result.cache_read_tokens = getattr(response.usage, 'cache_read_tokens', 0)
return response2.3 关键 Grafana 告警规则
yaml
# alerting-rules.yml
groups:
- name: llm_reliability
rules:
# 错误率突增
- alert: LLMHighErrorRate
expr: |
rate(llm_errors_total[5m]) / rate(llm_requests_total[5m]) > 0.05
for: 2m
labels:
severity: warning
annotations:
summary: "LLM error rate {{ $value | humanizePercentage }} on {{ $labels.model }}"
# TTFT 劣化
- alert: LLMHighTTFT
expr: |
histogram_quantile(0.95,
rate(llm_time_to_first_token_seconds_bucket[10m])
) > 5
for: 5m
annotations:
summary: "P95 TTFT > 5s on {{ $labels.model }}/{{ $labels.provider }}"
# 成本异常（按小时 rolling）
- alert: LLMCostAnomaly
expr: |
increase(llm_cost_usd_total[1h]) >
2 * avg_over_time(increase(llm_cost_usd_total[1h])[7d:1h])
for: 10m
labels:
severity: critical
annotations:
summary: "LLM hourly cost 2x above 7d average"
# 队列积压
- alert: LLMQueueDepthHigh
expr: llm_queue_depth > 100
for: 1m
annotations:
summary: "LLM queue depth {{ $value }} on {{ $labels.model }}"三、Logs：结构化、分层、不要把上下文塞进单条记录
3.1 日志分层策略
LLM 日志最大的问题是大小失控。一个包含 50K token 上下文的请求，如果你把 messages 直接 json-dump，单条日志会有 200KB+。这让 ELK 的 index size 爆炸，查询变慢，成本翻倍。
解决方案是日志分层：
vbscript
Level 1 - Request Log（每次请求一条）
├── 请求元数据：trace_id, model, provider, feature, tenant_id
├── 摘要指标：duration_ms, input_tokens, output_tokens, cost_usd
├── 状态：success/error, error_type
└── 内容引用：context_ref（指向 Level 2）
Level 2 - Context Log（按需，不进主索引）
├── system_prompt_hash + system_prompt（可选，按配置决定是否存全文）
├── messages_summary（每条消息的 role + token_count + 前100字）
└── tool_results_summary
Level 3 - Full Payload（可选，仅调试/审计场景）
├── full_messages（完整内容）
└── raw_response
python
# structured_logger.py
import json
import hashlib
import time
from typing import Any, Optional
import structlog
log = structlog.get_logger()
class LLMStructuredLogger:
def __init__(self, context_store=None, full_payload_store=None):
self.context_store = context_store
self.full_payload_enabled = full_payload_store is not None
self.full_payload_store = full_payload_store
def _compute_prompt_hash(self, messages: list) -> str:
"""只 hash system prompt（稳定部分），用于去重和缓存命中分析"""
system_content = ""
for msg in messages:
if isinstance(msg, dict) and msg.get("role") == "system":
system_content = str(msg.get("content", ""))
break
return hashlib.sha256(system_content.encode()).hexdigest()[:16]
def _summarize_messages(self, messages: list) -> list:
"""每条消息只保留 role + token 估算 + 前 100 字"""
summary = []
for msg in messages:
if not isinstance(msg, dict):
continue
content = msg.get("content", "")
if isinstance(content, list):
text_parts = [
p.get("text", "")[:100]
for p in content
if isinstance(p, dict) and p.get("type") == "text"
]
content_preview = " | ".join(text_parts)
has_images = any(
p.get("type") == "image"
for p in content
if isinstance(p, dict)
)
else:
content_preview = str(content)[:100]
has_images = False
summary.append({
"role": msg.get("role"),
"content_preview": content_preview,
"has_images": has_images,
"approx_tokens": len(str(content)) // 4,
})
return summary
def log_request(
self,
trace_id: str,
span_id: str,
request_id: str,
model: str,
provider: str,
feature: str,
tenant_id: str,
messages: list,
response: Any,
duration_ms: float,
input_tokens: int,
output_tokens: int,
cache_read_tokens: int,
cost_usd: float,
status: str,
error_type: Optional[str] = None,
error_message: Optional[str] = None,
extra: Optional[dict] = None,
):
prompt_hash = self._compute_prompt_hash(messages)
# Level 1: Request Log（进主日志索引）
request_log = {
"event": "llm_request",
"trace_id": trace_id,
"span_id": span_id,
"request_id": request_id,
"model": model,
"provider": provider,
"feature": feature,
"tenant_id": tenant_id,
"duration_ms": round(duration_ms, 2),
"tokens": {
"input": input_tokens,
"output": output_tokens,
"cache_read": cache_read_tokens,
"total": input_tokens + output_tokens,
},
"cost_usd": round(cost_usd, 6),
"status": status,
"prompt_hash": prompt_hash,
"message_count": len(messages),
"timestamp": time.time(),
}
if error_type:
request_log["error_type"] = error_type
if error_message:
request_log["error_message"] = str(error_message)[:500]
if extra:
request_log["extra"] = extra
log.info("llm_request", **request_log)
# Level 2: Context Log（写到独立存储，不进主索引）
if self.context_store:
context_log = {
"request_id": request_id,
"trace_id": trace_id,
"prompt_hash": prompt_hash,
"messages_summary": self._summarize_messages(messages),
"response_preview": self._extract_response_preview(response),
}
self.context_store.put(f"context/{request_id}.json", context_log)
# Level 3: Full Payload（仅 debug 模式）
if self.full_payload_enabled:
self.full_payload_store.put(
f"full/{request_id}.json.gz",
{"messages": messages, "response": response}
)
def _extract_response_preview(self, response: Any) -> str:
if not response:
return ""
if isinstance(response, dict):
content = response.get("content", "")
elif hasattr(response, "content"):
content = response.content
else:
content = str(response)
if isinstance(content, list):
texts = [
b.get("text", "") if isinstance(b, dict) else getattr(b, "text", "")
for b in content
if (isinstance(b, dict) and b.get("type") == "text") or
(hasattr(b, "type") and b.type == "text")
]
content = " ".join(texts)
return str(content)[:200]3.2 日志采样策略
生产环境下，不可能每条请求都打 Level 2/3 日志。正确的采样策略：
python
# log_sampler.py
import random
from enum import Enum
class LogLevel(Enum):
REQUEST_ONLY = 1 # 只打 Level 1
WITH_CONTEXT = 2 # Level 1 + 2
FULL_PAYLOAD = 3 # Level 1 + 2 + 3
class LLMLogSampler:
def __init__(
self,
error_sample_rate: float = 1.0,
slow_request_threshold_ms: float = 5000,
slow_request_sample_rate: float = 1.0,
normal_context_sample_rate: float = 0.01,
debug_full_payload_rate: float = 0.001,
force_trace_ids: set = None,
):
self.error_sample_rate = error_sample_rate
self.slow_threshold = slow_request_threshold_ms
self.slow_rate = slow_request_sample_rate
self.normal_context_rate = normal_context_sample_rate
self.debug_rate = debug_full_payload_rate
self.force_trace_ids = force_trace_ids or set()
def decide_log_level(
self,
trace_id: str,
status: str,
duration_ms: float,
) -> LogLevel:
if trace_id in self.force_trace_ids:
return LogLevel.FULL_PAYLOAD
if status == "error":
if random.random() < self.error_sample_rate:
return LogLevel.WITH_CONTEXT
return LogLevel.REQUEST_ONLY
if duration_ms > self.slow_threshold:
if random.random() < self.slow_rate:
return LogLevel.WITH_CONTEXT
r = random.random()
if r < self.debug_rate:
return LogLevel.FULL_PAYLOAD
if r < self.normal_context_rate:
return LogLevel.WITH_CONTEXT
return LogLevel.REQUEST_ONLY3.3 敏感信息的结构化脱敏
python
# log_redactor.py
import re
from typing import Any
class LLMLogRedactor:
PATTERNS = [
(re.compile(r'(sk-[a-zA-Z0-9]{20,})', re.IGNORECASE), '[API_KEY]'),
(re.compile(r'(Bearer\s+[a-zA-Z0-9\-_.]+)', re.IGNORECASE), 'Bearer [TOKEN]'),
(re.compile(r'\b1[3-9]\d{9}\b'), '[PHONE]'),
(re.compile(r'\b[a-zA-Z0-9._%+\-]+@[a-zA-Z0-9.\-]+\.[a-zA-Z]{2,}\b'), '[EMAIL]'),
(re.compile(r'\b\d{17}[\dXx]\b'), '[ID_CARD]'),
(re.compile(r'\b\d{16,19}\b'), '[BANK_CARD]'),
]
def redact(self, text: str) -> str:
if not isinstance(text, str):
return text
for pattern, replacement in self.PATTERNS:
text = pattern.sub(replacement, text)
return text
def redact_messages_preview(self, messages_summary: list) -> list:
result = []
for msg in messages_summary:
redacted = dict(msg)
if "content_preview" in redacted:
redacted["content_preview"] = self.redact(redacted["content_preview"])
result.append(redacted)
return result四、Traces：让每次 LLM 调用都可追溯、可归因
4.1 为什么 LLM Trace 和普通 HTTP Trace 不一样
普通 Trace：HTTP Request → DB Query → Cache Read → Response
LLM Trace 的现实：
sql
HTTP Request
├── Auth Check
├── Rate Limit Check
├── Context Assembly
│ ├── System Prompt Rendering (template + variables)
│ ├── Memory Retrieval (vector search)
│ └── Tool Results Injection
├── LLM Request (可能是流式，可能触发工具调用)
│ ├── Tool Call: search_web
│ │ └── HTTP → External API
│ ├── Tool Call: query_database
│ │ └── DB Query
│ └── Final Response Generation
├── Response Validation (JSON Schema check)
├── Post-processing (Guardrail, Redaction)
└── Streaming Response to Client挑战：
- 流式请求的 Span 结束时间不确定：不知道什么时候是"最后一个 token"
- 工具调用是嵌套的：一个 LLM 请求可能触发多轮工具调用，每轮又是新的子 Span
- Prompt 内容本身是诊断数据：可能很大（50K tokens），不能直接塞进 Span attributes
- 跨模型调用链：多个模型调用需要串联成一个 Trace
4.2 用 OpenTelemetry 构建 LLM Trace
python
# llm_tracer.py
from opentelemetry import trace
from opentelemetry.trace import SpanKind, Status, StatusCode
import time
tracer = trace.get_tracer("llm-service", "1.0.0")
class LLMSpanAttributes:
# GenAI 标准属性（OTel GenAI Semantic Conventions）
SYSTEM = "gen_ai.system"
REQUEST_MODEL = "gen_ai.request.model"
RESPONSE_MODEL = "gen_ai.response.model"
REQUEST_MAX_TOKENS = "gen_ai.request.max_tokens"
REQUEST_TEMPERATURE = "gen_ai.request.temperature"
USAGE_INPUT_TOKENS = "gen_ai.usage.input_tokens"
USAGE_OUTPUT_TOKENS = "gen_ai.usage.output_tokens"
# 扩展属性
CACHE_READ_TOKENS = "gen_ai.usage.cache_read_input_tokens"
TTFT_MS = "gen_ai.performance.time_to_first_token_ms"
TPOT_MS = "gen_ai.performance.time_per_output_token_ms"
COST_USD = "gen_ai.cost.estimated_usd"
FEATURE = "gen_ai.feature"
TENANT_ID = "gen_ai.tenant_id"
PROMPT_HASH = "gen_ai.prompt_hash"
CONTEXT_TOKENS = "gen_ai.context.total_tokens"
TOOL_CALL_COUNT = "gen_ai.tool_call_count"
FINISH_REASON = "gen_ai.response.finish_reason"
async def traced_llm_call(
messages: list,
model: str,
provider: str,
feature: str,
tenant_id: str,
temperature: float = 0.7,
max_tokens: int = 4096,
tools: list = None,
):
span_name = f"gen_ai.{provider}.{model}"
with tracer.start_as_current_span(
span_name,
kind=SpanKind.CLIENT,
) as span:
span.set_attribute(LLMSpanAttributes.SYSTEM, provider)
span.set_attribute(LLMSpanAttributes.REQUEST_MODEL, model)
span.set_attribute(LLMSpanAttributes.REQUEST_MAX_TOKENS, max_tokens)
span.set_attribute(LLMSpanAttributes.REQUEST_TEMPERATURE, temperature)
span.set_attribute(LLMSpanAttributes.FEATURE, feature)
span.set_attribute(LLMSpanAttributes.TENANT_ID, tenant_id)
total_context_tokens = sum(
len(str(m.get("content", ""))) // 4
for m in messages
)
span.set_attribute(LLMSpanAttributes.CONTEXT_TOKENS, total_context_tokens)
span.set_attribute(LLMSpanAttributes.PROMPT_HASH, _hash_system_prompt(messages))
request_start = time.monotonic()
first_token_time = None
tool_calls_made = 0
try:
response = await _do_stream_call(
messages=messages, model=model,
temperature=temperature, max_tokens=max_tokens, tools=tools,
)
async for event in response.stream:
if hasattr(event, 'type'):
if event.type == "content_block_start" and first_token_time is None:
first_token_time = time.monotonic()
if event.type == "content_block_start":
if hasattr(event, 'content_block') and \
getattr(event.content_block, 'type', None) == "tool_use":
tool_calls_made += 1
final = response.get_final_message()
duration_ms = (time.monotonic() - request_start) * 1000
if first_token_time:
ttft = (first_token_time - request_start) * 1000
span.set_attribute(LLMSpanAttributes.TTFT_MS, round(ttft, 2))
if final.usage.output_tokens > 1:
tpot = (duration_ms - ttft) / (final.usage.output_tokens - 1)
span.set_attribute(LLMSpanAttributes.TPOT_MS, round(tpot, 2))
span.set_attribute(LLMSpanAttributes.USAGE_INPUT_TOKENS, final.usage.input_tokens)
span.set_attribute(LLMSpanAttributes.USAGE_OUTPUT_TOKENS, final.usage.output_tokens)
cache_read = getattr(final.usage, 'cache_read_input_tokens', 0)
if cache_read:
span.set_attribute(LLMSpanAttributes.CACHE_READ_TOKENS, cache_read)
if tools:
span.set_attribute(LLMSpanAttributes.TOOL_CALL_COUNT, tool_calls_made)
finish_reason = getattr(final, 'stop_reason', 'unknown') or "unknown"
span.set_attribute(LLMSpanAttributes.FINISH_REASON, finish_reason)
span.set_status(Status(StatusCode.OK))
return final
except Exception as e:
span.set_status(Status(StatusCode.ERROR, str(e)))
span.record_exception(e)
raise
def _hash_system_prompt(messages: list) -> str:
import hashlib
for msg in messages:
if isinstance(msg, dict) and msg.get("role") == "system":
return hashlib.sha256(
str(msg.get("content", "")).encode()
).hexdigest()[:16]
return "no_system_prompt"4.3 Trace 上下文跨服务传播
python
# trace_propagation.py
from opentelemetry import trace, propagate
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
propagator = TraceContextTextMapPropagator()
def inject_trace_context(headers: dict) -> dict:
propagate.inject(headers)
return headers
# FastAPI 中间件示例
@app.middleware("http")
async def trace_middleware(request, call_next):
ctx = propagate.extract(dict(request.headers))
with tracer.start_as_current_span(
f"http.{request.method} {request.url.path}",
context=ctx,
kind=SpanKind.SERVER,
) as span:
span.set_attribute("http.method", request.method)
span.set_attribute("http.url", str(request.url))
response = await call_next(request)
span.set_attribute("http.status_code", response.status_code)
return response五、三件套的集成：让 Metrics、Logs、Traces 互相关联
5.1 关联字段设计
| 字段 | 说明 | 示例 |
|---|---|---|
trace_id |
OTel trace ID，16字节 hex | 4bf92f3577b34da6... |
span_id |
当前 Span ID | 00f067aa0ba902b7 |
request_id |
业务层请求 ID | req_abc123 |
model |
使用的模型名称 | deepseek-v3 |
feature |
业务功能标识 | code_review |
tenant_id |
租户标识 | org_xyz |
prompt_hash |
System prompt 的哈希 | a3f8c2d1 |
5.2 Exemplar：从 Metric 跳转到 Trace
python
# exemplar_integration.py
from prometheus_client import Histogram
from opentelemetry import trace as otel_trace
llm_duration_with_exemplar = Histogram(
'llm_request_duration_with_exemplar_seconds',
'Duration with exemplar support',
['model', 'provider'],
)
def record_duration_with_exemplar(model: str, provider: str, duration: float):
current_span = otel_trace.get_current_span()
span_context = current_span.get_span_context()
exemplar = None
if span_context.is_valid:
trace_id_hex = format(span_context.trace_id, '032x')
exemplar = {"traceID": trace_id_hex}
llm_duration_with_exemplar.labels(
model=model, provider=provider
).observe(duration, exemplar=exemplar)5.3 Log 中嵌入 Trace ID
python
# log_trace_integration.py
import structlog
from opentelemetry import trace
def add_trace_context(logger, method, event_dict):
span = trace.get_current_span()
ctx = span.get_span_context()
if ctx.is_valid:
event_dict["trace_id"] = format(ctx.trace_id, '032x')
event_dict["span_id"] = format(ctx.span_id, '016x')
event_dict["trace_flags"] = ctx.trace_flags
return event_dict
structlog.configure(
processors=[
structlog.stdlib.add_log_level,
structlog.stdlib.add_logger_name,
add_trace_context,
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.JSONRenderer(),
]
)六、生产落地：成本与复杂度的平衡
6.1 Observability 的隐性成本
| 数据类型 | 保留策略 | 存储位置 |
|---|---|---|
| Level 1 Request Log | 30天热存储 + 90天冷存储 | ELK → OSS/COS |
| Level 2 Context Log | 7天热存储（1% 采样） | ELK |
| Level 3 Full Payload | 24小时（0.1% 采样） | 对象存储 |
| Trace（正常） | 3天（5% 采样） | Tempo/Jaeger |
| Trace（错误/慢请求） | 14天（100%） | Tempo/Jaeger |
| Metrics | 15天高分辨率 + 1年降采样 | Prometheus + Thanos |
6.2 渐进式落地路线图
Phase 1（Day 1--7）：基础可见性
- 接入 Level 1 Request Log（只有 metadata，没有内容）
- 暴露 5 个核心 Metrics：请求数、错误率、P95 延迟、token 消耗、成本
- 最简单的 Trace：只 trace 整个 LLM 请求
- 目标：能回答"现在有多少请求失败？"
Phase 2（Week 2--4）：诊断能力
- 加 TTFT/TPOT 指标
- 加 Level 2 Context Log（1% 采样）
- 加工具调用子 Span
- 加 Exemplar（metric → trace 跳转）
- 目标：能回答"慢请求是卡在哪里？"
Phase 3（Month 2+）：运营智能
- 加 prompt_hash 分析（哪些 system prompt 导致了更高错误率）
- 加 tenant_id 维度（哪个租户在烧钱）
- 加 token 效率趋势
- 接入 LLM Eval 信号
- 目标：能回答"上周上线的新 Prompt 有没有让质量变差？"
6.3 常见坑
坑 1：用同一套 bucket 分布看所有 LLM 请求延迟
不同功能延迟分布差异极大。实时聊天（P95 目标 2s）和批量分析（P95 可以接受 60s）混在一起，告警阈值无法合理设置。解决：按 feature label 分开看延迟分布，每个功能有独立 SLO。
坑 2：把 model name 硬编码在 metric label 里
模型版本升级后，所有历史 metric 断开。解决：在应用层做 model alias 映射，metric label 用 alias，不用具体版本号。
坑 3：Trace 里存了原始 messages，被安全审计发现
生产 Trace 里不能有用户 PII。解决：只存 prompt_hash + 消息摘要，原始内容走独立的审计日志（加密存储，访问受控）。
坑 4：流式请求 Span 生命周期管理
流式请求可能跨越 30 秒，Span 开着期间 Jaeger/Tempo 不会索引。对于流式请求，额外发一个 stream_start event 到 Prometheus（带 trace_id）用于实时监控，不依赖 Trace 系统做实时告警。
七、总结
LLM Observability 不是把现有监控体系套到 AI 上，而是在三件套基础上补充 LLM 特有的语义：
- Metrics：除了延迟/错误率，增加 TTFT、TPOT、token 消耗（分 input/output/cache）、成本归因。按 feature + tenant_id 分维度，每个功能有独立 SLO。
- Logs：分三层，Level 1 进索引，Level 2 采样进独立存储，Level 3 仅 debug。Prompt 内容不进主日志，用 prompt_hash + 摘要代替全文。上线脱敏管道，过滤 PII。
- Traces：用 OpenTelemetry GenAI Semantic Conventions。工具调用是子 Span。Span attributes 存 prompt_hash，不存原文。用 Exemplar 打通 metric → trace 跳转。Trace context 跨服务传播。
三件套的价值在于关联：同一个 trace_id 贯穿 Metric Exemplar、Log 记录和 Trace Span，出了问题 5 分钟内就能定位到是哪个租户、哪个 feature、哪个 Prompt，而不是靠猜。
渐进式落地：Phase 1 先让 5 个核心 metric 可见，Phase 2 补诊断能力，Phase 3 做运营智能。Observability 本身的成本控制和数据治理，也是生产工程的一部分。