【架构实战】OpenTelemetry链路追踪实战：从零到生产的完整指南

【架构实战】OpenTelemetry链路追踪实战：从零到生产的完整指南

上篇文章我们聊了可观测性的三大支柱------Metrics、Logs、Traces。很多同学说"道理懂了，代码怎么写还是一头雾水"。今天就来手把手实战，用OpenTelemetry从零搭建一套生产级的链路追踪系统。

一、为什么选OpenTelemetry？

市面上的APM方案很多（Jaeger、Zipkin、SkyWalking），但OpenTelemetry（简称OTel）是CNCF唯一的可观测性标准，它有三个优势：

• 厂商无关：今天用Jaeger，明天换Pinpoint，改配置就行不用改代码
• 多语言统一：Java、Go、Python、Node.js全支持，一套协议走天下
• 自动+手动：零侵入自动埋点 + 按需手动埋点，灵活可控

二、环境准备

我们用Docker Compose一键启动所有组件：

version: '3.8'
services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"   # gRPC
      - "4318:4318"   # HTTP
      - "8888:8888"   # Prometheus metrics
      - "8889:8889"   # Prometheus exporter metrics

  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"  # UI
      - "14250:14250"  # gRPC
    environment:
      - COLLECTOR_OTLP_ENABLED=true

  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning

Collector配置文件：

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  memory_limiter:
    check_interval: 2s
    limit_percentage: 80

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true
  prometheus:
    endpoint: "0.0.0.0:8889"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [jaeger]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheus]

三、Python服务埋点

我们模拟一个典型的微服务调用链：API Gateway → User Service → Order Service → Product Service。

3.1 安装依赖

pip install opentelemetry-api \
            opentelemetry-sdk \
            opentelemetry-exporter-otlp \
            opentelemetry-instrumentation-flask \
            opentelemetry-instrumentation-requests \
            opentelemetry-instrumentation-redis

3.2 初始化SDK

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource, SERVICE_NAME

def init_tracing(service_name: str, otlp_endpoint: str = "http://localhost:4317"):
    resource = Resource.create({
        SERVICE_NAME: service_name,
        "service.version": "1.0.0",
        "deployment.environment": "production"
    })

    provider = TracerProvider(resource=resource)
    exporter = OTLPSpanExporter(endpoint=otlp_endpoint, insecure=True)
    provider.add_span_processor(BatchSpanProcessor(exporter))

    trace.set_tracer_provider(provider)
    return trace.get_tracer(service_name)

3.3 手动埋点

自动埋点只能覆盖HTTP和数据库，手动埋点才是精髓：

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = init_tracing("order-service")

def create_order(order_id: str, user_id: str):
    with tracer.start_as_current_span("create_order") as span:
        # 设置关键属性
        span.set_attribute("order.id", order_id)
        span.set_attribute("order.user_id", user_id)

        try:
            # 验证库存
            with tracer.start_as_current_span("check_inventory") as child:
                child.set_attribute("product.count", 10)
                inventory = check_inventory_db(order_id)
                child.set_attribute("inventory.available", inventory)

            # 创建订单记录
            with tracer.start_as_current_span("save_order"):
                order = save_order_to_db(order_id, user_id)
                span.set_attribute("order.total_amount", order.total)

            # 发送通知（异步，不阻塞主流程）
            with tracer.start_as_current_span("send_notification", kind=SpanKind.PRODUCER):
                publish_event("order_created", order)

            span.set_status(Status(StatusCode.OK))
            return order

        except InsufficientStockError as e:
            span.set_status(Status(StatusCode.ERROR, str(e)))
            span.record_exception(e)
            raise

3.4 跨服务上下文传播

这是链路追踪最难的部分------Trace如何在服务间传递？

# 服务A：注入TraceContext到HTTP Header
from opentelemetry.propagate import inject, extract

def call_downstream_service(url: str, data: dict):
    headers = {}
    inject(headers)  # 自动把当前TraceContext注入到headers

    response = requests.post(
        url,
        json=data,
        headers={**headers, "Content-Type": "application/json"}
    )
    return response.json()

# 服务B：从Header提取TraceContext
from opentelemetry.propagate import extract
from opentelemetry.trace import SpanKind

def handle_request(request):
    context = extract(request.headers)  # 从incoming请求中提取

    with tracer.start_as_current_span(
        "handle_request",
        context=context,
        kind=SpanKind.SERVER
    ) as span:
        span.set_attribute("http.method", request.method)
        span.set_attribute("http.url", request.url)
        # 业务逻辑...

四、Docker Compose一键启动

# 启动所有组件
docker-compose up -d

# 验证服务状态
docker-compose ps

# 查看Jaeger UI
# 浏览器访问 http://localhost:16686

五、Grafana大盘配置

光有Trace不够，还要和Metrics联动。创建一个链路追踪大盘：

{
  "panels": [
    {
      "title": "请求量 Top 10 服务",
      "type": "bargauge",
      "targets": [{
        "expr": "topk(10, sum by (service_name) (rate(otelcol_exporter_sent_spans{type=\"traces\"}[5m])))"
      }]
    },
    {
      "title": "P99延迟分布",
      "type": "heatmap",
      "targets": [{
        "expr": "histogram_quantile(0.99, sum by (service_name, le) (rate(otelgrpc_io_server_duration_bucket[5m])))"
      }]
    },
    {
      "title": "错误率热力图",
      "type": "stat",
      "targets": [{
        "expr": "sum by (service_name) (rate(otel_exporter_sent_spans{status_code=\"ERROR\"}[5m])) / sum by (service_name) (rate(otel_exporter_sent_spans_total[5m])) * 100"
      }]
    }
  ]
}

六、生产经验总结

跑了半年，总结几条避坑指南：

1. 采样策略决定成本

全量采样在生产环境是灾难。推荐：

• 固定采样：只保留10%的请求
• 尾部采样：所有错误请求+超过P99的慢请求必须保留

sampler = TraceIdRatioBased(0.1)  # 10%采样率

2. 属性命名规范

提前制定命名规范，避免"order_id"、"orderId"、"orderid"混用：

{attribute}.{sub_attribute}
例：order.total_amount, user.id, product.sku

3. 不要在Span里放敏感信息

信用卡号、密码、完整身份证号不要出现在Attributes里，那是给监控用的不是日志用的。

4. 异步任务的Trace断开问题

Celery/Bull队列里的任务，默认会断开Trace。解决方案是用context.propagate()把Context序列化后传到队列里。

七、效果验证

用JMeter或wrk打个压测：

# 模拟100并发，持续30秒
wrk -t4 -c100 -d30s http://localhost:8080/api/orders

然后去Jaeger里搜索：

• 输入服务名过滤：service = "order-service"
• 查看耗时最长的Trace：选择排序方式为Duration
• 点击任意一个Trace，看完整的调用瀑布图

你会看到：Gateway收到请求（0ms）→ UserService鉴权（3ms）→ OrderService创建订单（45ms）→ ProductService查库存（8ms）→ 回调通知（12ms），总耗时68ms，每一个环节清晰可见。

---

下期预告：链路追踪只是起点，如何把这些数据真正用起来？下一期聊聊如何用Grafana+OpenTelemetry搭建全链路可观测平台，以及如何设置智能告警------不是"CPU>80%"这种粗放告警，而是"XX接口P99延迟环比增长30%"的精准告警。