04 Touring the Stack: One Checkout, Traced Through Every Container

Touring the Stack: One Checkout, Traced Through Every Container

Part 4 of a series on observability for microservices. Part 3 built the stack; this post drives it. Every command below is copy-pasteable against the companion repo's stack/ once it's running. Series index.

📦 GitHub: https://github.com/geekchow/O11y-Micro-Service

Reading a pipeline diagram only gets you so far. This post pushes one HTTP request through a live 13-container stack and finds its footprint in every component --- trace, metrics, and logs --- with real commands.

We'll deliberately trigger a declined checkout : any amountCents ending in 7 gets rejected by the payment service. Error traces are ideal tour subjects because the Collector's tail-sampling policy always keeps them --- there's no chance our request gets sampled away before we can find it.

Step 0 --- send the request

bash 复制代码
curl -s -X POST localhost:8080/checkout -H 'Content-Type: application/json' \
  -d '{"cartId":"tour","amountCents":47,"tenant":"acme"}'
# → {"orderId":"<UUID>","status":"declined"}     amount ends in 7 ⇒ declined

Note the orderId in the response --- it's our join key into the logs, from which we'll pull the trace_id.

Step 1 --- what the JVMs just did, invisibly

The five services already did their work without any code watching: Tomcat and RestClient auto-instrumentation created spans, the manual checkout span ran, and a WARN log line fired in both payment and gateway. There's nothing to see inside the services --- by design, that work happens off to the side, buffered and shipped asynchronously so it never blocks the response the customer sees.

Step 2 --- Loki has the trace_id first

bash 复制代码
ORDER=<paste the orderId>
curl -s -G 'localhost:3100/loki/api/v1/query_range' \
  --data-urlencode "query={service_name=\"payment\"} |= \"$ORDER\"" \
  --data-urlencode 'limit=1' \
  | python3 -c "import json,sys; s=json.load(sys.stdin)['data']['result'][0]['stream']; print('trace_id:', s['trace_id'])"

The trace_id comes back as structured metadata , not as text inside the log line --- the OTel Logback appender stamped it because the WARN happened inside the span's active Context. This is the mechanism, not a coincidence: log.warn(...) never mentions trace IDs anywhere in the application code.

Step 3 --- Tempo has the whole story

bash 复制代码
TID=<paste the trace_id>
curl -s "localhost:3200/api/traces/$TID" | python3 -c "
import json,sys
t=json.load(sys.stdin)
for b in t['batches']:
    svc=[a['value']['stringValue'] for a in b['resource']['attributes'] if a['key']=='service.name'][0]
    for ss in b.get('scopeSpans',[]):
        for s in ss['spans']: print(f\"{svc:10s} {s['name']}\")"

Expect spans from four services --- gateway (including the manual checkout span), auth, cart → inventory, and payment --- all under one trace_id. This is context propagation, made visible after the fact. This trace exists in Tempo because the Collector's status_code: ERROR policy matched it; a healthy sibling checkout only has a 25% chance of surviving the same tail-sampling filter in this demo config (1% in a production-like ratio).

Step 4 --- it's already a number in Prometheus

bash 复制代码
open 'http://localhost:9090/graph?g0.expr=sum%20by%20(service_name%2C%20status_code)%20(rate(traces_span_metrics_calls_total%7Bspan_kind%3D%22SPAN_KIND_SERVER%22%7D%5B2m%5D))'

Your checkout is one anonymous increment inside traces_span_metrics_calls_total{status_code="STATUS_CODE_ERROR"}, produced by the spanmetrics connector from the same spans you just saw in Tempo --- but generated before tail sampling ran. Individual identity is gone; the rate is preserved. That's the metrics trade-off, mechanized in one connector.

Step 5 --- watch the pipeline count your spans

bash 复制代码
# agent tier accepted them...
curl -s localhost:8888/metrics | grep -E '^otelcol_receiver_accepted_spans'
# ...gateway tier accepted them, and exported FEWER than it accepted:
curl -s localhost:8889/metrics | grep -E '^otelcol_(receiver_accepted|exporter_sent)_spans'

The gap between the gateway's accepted and sent span counts is the tail sampler discarding healthy traces --- the only place in the whole stack where telemetry is deliberately thrown away. Watch it live at the zpages debug endpoint: http://localhost:55680/debug/pipelinez.

Step 6 --- close the loop in Grafana

Open http://localhost:3000 → Checkout --- RED dashboard:

  1. Error-rate panel --- your decline shows up in the red series (the metric view).
  2. p99 panel → click an exemplar dot --- the trace opens directly (metric → trace pivot).
  3. In the waterfall, click the payment span → "Logs for this span" --- your WARN line appears (trace → log pivot).
  4. In the log line's details, click trace_id → "View trace" --- you're back at the trace (log → trace pivot --- full circle).

None of those three pivots involve a plugin or a glue service. Each one is a single provisioning key in Grafana's datasource config:

Click Wiring key What it does
exemplar dot → trace exemplarTraceIdDestinations on the Prometheus datasource reads the exemplar's trace_id label, opens it in Tempo
span → its logs tracesToLogsV2.query on the Tempo datasource templated Loki query: `{service_name=...}
log line → trace derivedFields on the Loki datasource lifts the trace_id metadata into a View trace link

The pivots exist because every signal carries the same trace_id --- Grafana just needs to be told which field holds it in each store.

What one request became

One HTTP call turned into: roughly ten spans converging in Tempo under a single trace_id, one increment across three Prometheus counters plus an exemplar pointing back at the trace, and two WARN lines in Loki carrying that same trace_id as structured metadata --- while both Collectors visibly counted, enriched, and (for other, healthier traces) culled data along the way. Every container did exactly the one job it owns, and you just watched each of them do it.

The next post takes this further: reproducing a real production incident on purpose, by flipping one environment variable, and watching the entire detection-to-rollback loop fire on your own machine.

➡️ Next: Part 5 --- Reproducing a Production Incident on Purpose

相关推荐
7177773 天前
支持混合云的 DevOps 平台有哪些:Gitee、Azure DevOps、GitLab、Jenkins、JFrog 横向对比
gitee·azure·devops
小南家的青蛙5 天前
微服务体系下DevOps运维岗位面试方案设计
运维·面试·职场和发展·devops
逐流人7 天前
手工建集群到全自动发布:k8s集群部署与DevOps流水线整合实战
ci/cd·云原生·kubernetes·云计算·devops
无所畏俱的老尹10 天前
Agent 落地为什么需要影子对照:同输入只读双跑、写动作变提案、差异报警,比直接切全量更安全
人工智能·机器学习·云计算·aigc·devops
pengdott10 天前
Rocky linux9安装最新版sonarqube-26.10.0
ci/cd·devops·sonarqube
逐流人10 天前
DevOps 入门实战(一):从 CI/CD 概念到 Jenkins 自动构建部署
运维·云原生·云计算·jenkins·devops
Lyy12 天前
DevOps平台 — 第十九篇:消息通知与预警的设计与实现
后端·devops·vibecoding
长不大的杨瑞杰13 天前
OpenCode v2 配置工程实战:Config 优先级、JSONC、变量替换与多环境治理
人工智能·安全·云计算·软件工程·devops
cakeism82513 天前
2026 中国企业 DevOps 平台选型:从工具链到一体化平台的决策框架
运维·servlet·devops
Zadig15 天前
从 Claude Code 投毒到币安入侵:高权限 DevOps 终端,正在成为黑客突破口
安全·devops