08 Inside the Collector: Pipelines, Deployment Patterns, and Failure Modes

Inside the Collector: Pipelines, Deployment Patterns, and Failure Modes

Part 8 of a series on observability for microservices. Part 7 covered signals and context propagation. This post covers the piece that moves telemetry once it leaves your process: the OpenTelemetry Collector. Series index.

📦 GitHub: https://github.com/geekchow/O11y-Micro-Service

What it is, in one line

The Collector is a single Go binary that runs configurable receivers → processors → exporters pipelines --- the vendor-agnostic middle tier that lets apps offload telemetry fast and lets backends become a config choice instead of a code change.

Could your app export straight to a backend, skipping the Collector entirely? Yes --- for local dev or a tiny setup, that's fine. Production wants the middle tier because it centralizes everything you must not scatter across fifty services: retries, batching, credentials, PII scrubbing, sampling policy, and vendor routing.

The five component types

Type Role Workhorses you'll actually use
Receivers Data in (listen or scrape) otlp (gRPC :4317 / HTTP :4318); prometheus (scrapes targets!), filelog (tails files), jaeger, kafka, hostmetrics
Processors Transform in flight memory_limiter (always first), batch (always ~last), attributes/resource (add/drop/rename), filter, transform (OTTL language), k8sattributes (pod metadata enrichment), tail_sampling
Exporters Data out, with retry + persistent queue otlp/otlphttp, prometheusremotewrite, debug (to console), vendor exporters (splunk_hec, datadog...)
Connectors Exporter of pipeline A and receiver of pipeline B spanmetrics (spans → RED metrics), servicegraph, forward
Extensions Side services, touch no telemetry health_check, pprof, zpages (live pipeline debugging)

Config anatomy: define, then wire

The config has two halves, and the second half is the one people forget: defining a component does nothing until a pipeline references it.

yaml 复制代码
receivers:
  otlp:
    protocols: { grpc: {}, http: {} }
processors:
  memory_limiter: { check_interval: 1s, limit_percentage: 80 }
  batch: {}
exporters:
  otlphttp/tempo: { endpoint: http://tempo:4318 }
  prometheusremotewrite: { endpoint: http://mimir:9009/api/v1/push }
connectors:
  spanmetrics: {}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/tempo, spanmetrics]   # fan-out: both get every batch
    metrics:
      receivers: [otlp, spanmetrics]             # ← connector re-enters here
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]

#mermaid-svg-MJ3xpqjJ9ecar9r8{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-MJ3xpqjJ9ecar9r8 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .error-icon{fill:#552222;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .marker.cross{stroke:#333333;}#mermaid-svg-MJ3xpqjJ9ecar9r8 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-MJ3xpqjJ9ecar9r8 p{margin:0;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .cluster-label text{fill:#333;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .cluster-label span{color:#333;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .cluster-label span p{background-color:transparent;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .label text,#mermaid-svg-MJ3xpqjJ9ecar9r8 span{fill:#333;color:#333;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .node rect,#mermaid-svg-MJ3xpqjJ9ecar9r8 .node circle,#mermaid-svg-MJ3xpqjJ9ecar9r8 .node ellipse,#mermaid-svg-MJ3xpqjJ9ecar9r8 .node polygon,#mermaid-svg-MJ3xpqjJ9ecar9r8 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .rough-node .label text,#mermaid-svg-MJ3xpqjJ9ecar9r8 .node .label text,#mermaid-svg-MJ3xpqjJ9ecar9r8 .image-shape .label,#mermaid-svg-MJ3xpqjJ9ecar9r8 .icon-shape .label{text-anchor:middle;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .rough-node .label,#mermaid-svg-MJ3xpqjJ9ecar9r8 .node .label,#mermaid-svg-MJ3xpqjJ9ecar9r8 .image-shape .label,#mermaid-svg-MJ3xpqjJ9ecar9r8 .icon-shape .label{text-align:center;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .node.clickable{cursor:pointer;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .arrowheadPath{fill:#333333;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-MJ3xpqjJ9ecar9r8 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-MJ3xpqjJ9ecar9r8 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-MJ3xpqjJ9ecar9r8 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .cluster text{fill:#333;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .cluster span{color:#333;}#mermaid-svg-MJ3xpqjJ9ecar9r8 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-MJ3xpqjJ9ecar9r8 rect.text{fill:none;stroke-width:0;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .icon-shape,#mermaid-svg-MJ3xpqjJ9ecar9r8 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .icon-shape p,#mermaid-svg-MJ3xpqjJ9ecar9r8 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .icon-shape .label rect,#mermaid-svg-MJ3xpqjJ9ecar9r8 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-MJ3xpqjJ9ecar9r8 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-MJ3xpqjJ9ecar9r8 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} metrics pipeline
traces pipeline
otlp receiver
memory_limiter
batch
otlphttp → Tempo
spanmetrics connector
memory_limiter
batch
remote_write → Mimir
otlp receiver

Three rules of thumb worth internalizing: memory_limiter goes first (refuse before you swell), batch goes last before exporters (compress the network), and one pipeline per signal type --- with the name/instance syntax (otlphttp/tempo) when you need multiple instances of one component type, exactly as the config above does.

On distributions : the same engine ships in flavors --- core (minimal, curated), contrib (~everything; this is where tail_sampling, k8sattributes, and vendor exporters live, and the usual choice), k8s , and otlp-only . For production hardening, build a custom binary containing only the components you actually use with the OpenTelemetry Collector Builder (ocb). Component maturity varies individually per component --- check each one's README, not just the distribution as a whole.

A real gateway config, annotated

Here's the tail-sampling gateway Collector from the companion stack, with the parts that trip people up called out:

yaml 复制代码
connectors:
  spanmetrics:
    namespace: traces.span.metrics
    exemplars:
      enabled: true

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 256
    spike_limit_mib: 64
  batch: {}
  tail_sampling:
    decision_wait: 10s
    num_traces: 20000
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow
        type: latency
        latency: { threshold_ms: 2000 }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 25 }

service:
  pipelines:
    traces:                       # the culled stream → Tempo
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp/tempo]
    traces/spanmetrics:           # the UNCUT stream → RED metrics
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [spanmetrics]

That second pipeline --- traces/spanmetrics --- is not an accident or duplication. It reads the same incoming OTLP spans as the traces pipeline but skips tail_sampling entirely, because RED metrics need to see every request, not just the ones tail sampling decided to keep. Get this ordering backwards --- run spanmetrics after sampling --- and your dashboard's error rate silently understates reality by whatever fraction tail sampling discarded.

Deployment patterns --- the decision that actually matters

Pattern Topology Choose when
No collector SDK → backend directly Dev, demos, tiny setups
Agent SDK → Collector on the same node (DaemonSet/sidecar) → backend Default production baseline: apps offload in microseconds, agent adds node/pod metadata no central box could know
Gateway SDKs/agents → load-balanced central Collector tier → backends You need fleet-wide policy in one place: tail sampling, PII scrubbing, egress credentials, per-vendor routing
Agent + gateway Both The common end-state at scale

The trade is locality vs centrality: agents know node things and sit near the app; gateways see whole traces and hold one copy of policy and secrets. Tail sampling forces a gateway tier by definition --- a per-node agent can never see all the spans of a distributed trace, since those spans originate on different nodes. Part 9 covers the routing trick this requires when you scale the gateway tier past a single instance.

Failure modes --- how the pipeline sheds load

The Collector pipeline is a chain of queues, and it's worth knowing exactly how each link fails:
#mermaid-svg-9hYUciI7DCpuqMPX{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-9hYUciI7DCpuqMPX .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-9hYUciI7DCpuqMPX .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-9hYUciI7DCpuqMPX .error-icon{fill:#552222;}#mermaid-svg-9hYUciI7DCpuqMPX .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-9hYUciI7DCpuqMPX .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-9hYUciI7DCpuqMPX .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-9hYUciI7DCpuqMPX .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-9hYUciI7DCpuqMPX .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-9hYUciI7DCpuqMPX .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-9hYUciI7DCpuqMPX .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-9hYUciI7DCpuqMPX .marker{fill:#333333;stroke:#333333;}#mermaid-svg-9hYUciI7DCpuqMPX .marker.cross{stroke:#333333;}#mermaid-svg-9hYUciI7DCpuqMPX svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-9hYUciI7DCpuqMPX p{margin:0;}#mermaid-svg-9hYUciI7DCpuqMPX .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-9hYUciI7DCpuqMPX .cluster-label text{fill:#333;}#mermaid-svg-9hYUciI7DCpuqMPX .cluster-label span{color:#333;}#mermaid-svg-9hYUciI7DCpuqMPX .cluster-label span p{background-color:transparent;}#mermaid-svg-9hYUciI7DCpuqMPX .label text,#mermaid-svg-9hYUciI7DCpuqMPX span{fill:#333;color:#333;}#mermaid-svg-9hYUciI7DCpuqMPX .node rect,#mermaid-svg-9hYUciI7DCpuqMPX .node circle,#mermaid-svg-9hYUciI7DCpuqMPX .node ellipse,#mermaid-svg-9hYUciI7DCpuqMPX .node polygon,#mermaid-svg-9hYUciI7DCpuqMPX .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-9hYUciI7DCpuqMPX .rough-node .label text,#mermaid-svg-9hYUciI7DCpuqMPX .node .label text,#mermaid-svg-9hYUciI7DCpuqMPX .image-shape .label,#mermaid-svg-9hYUciI7DCpuqMPX .icon-shape .label{text-anchor:middle;}#mermaid-svg-9hYUciI7DCpuqMPX .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-9hYUciI7DCpuqMPX .rough-node .label,#mermaid-svg-9hYUciI7DCpuqMPX .node .label,#mermaid-svg-9hYUciI7DCpuqMPX .image-shape .label,#mermaid-svg-9hYUciI7DCpuqMPX .icon-shape .label{text-align:center;}#mermaid-svg-9hYUciI7DCpuqMPX .node.clickable{cursor:pointer;}#mermaid-svg-9hYUciI7DCpuqMPX .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-9hYUciI7DCpuqMPX .arrowheadPath{fill:#333333;}#mermaid-svg-9hYUciI7DCpuqMPX .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-9hYUciI7DCpuqMPX .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-9hYUciI7DCpuqMPX .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-9hYUciI7DCpuqMPX .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-9hYUciI7DCpuqMPX .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-9hYUciI7DCpuqMPX .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-9hYUciI7DCpuqMPX .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-9hYUciI7DCpuqMPX .cluster text{fill:#333;}#mermaid-svg-9hYUciI7DCpuqMPX .cluster span{color:#333;}#mermaid-svg-9hYUciI7DCpuqMPX div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-9hYUciI7DCpuqMPX .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-9hYUciI7DCpuqMPX rect.text{fill:none;stroke-width:0;}#mermaid-svg-9hYUciI7DCpuqMPX .icon-shape,#mermaid-svg-9hYUciI7DCpuqMPX .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-9hYUciI7DCpuqMPX .icon-shape p,#mermaid-svg-9hYUciI7DCpuqMPX .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-9hYUciI7DCpuqMPX .icon-shape .label rect,#mermaid-svg-9hYUciI7DCpuqMPX .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-9hYUciI7DCpuqMPX .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-9hYUciI7DCpuqMPX .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-9hYUciI7DCpuqMPX :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} )|OTLP| RCVreceiver

RCV
backpressure
queue full → drop

this exporter only
refusal propagates as

OTLP error to SDK
SDK BatchSpanProcessor

bounded queue → drops
memory_limiter

refuses batches under pressure
processors
exporter queue

  • retry w/ backoff

optional file_storage persistence
backend

down or slow
dropped

  • Backend down: that exporter's queue absorbs the burst, retries with backoff, and eventually drops --- but other exporters in the same fan-out keep flowing normally. This is per-backend blast-radius isolation. Add the file_storage extension for a queue that survives a Collector restart.
  • Collector overwhelmed: memory_limiter starts refusing incoming batches; the refusal propagates back as an OTLP error to the SDK; the SDK's own queue fills and drops at the source. Every stage degrades gracefully rather than blocking the application thread or OOM-killing the Collector.
  • Collector down entirely: the SDK retries briefly, then drops. This is exactly why the agent tier runs on-node --- there's almost nothing to lose on that first hop --- and why the Collector's own health_check and internal metrics feed back into Prometheus. It's the "watcher's watcher" pattern applied to the observability pipeline itself.

Given any Collector YAML, you should now be able to draw its pipelines on a whiteboard, argue agent-vs-gateway for a given org's needs, and, for any failure scenario ("Splunk ingest is lagging"), say exactly which queue fills and what gets dropped.

Next: the policy that the gateway tier exists to run --- sampling.

➡️ Next: Part 9 --- Sampling: Keeping the Interesting 1%

相关推荐
JonaHu17 小时前
AceDataCloud Coding Plan 多模型编程:本地、CI 与团队仓库的 Token 隔离实战
ai编程·devops·codex·api安全·claude code
吴建旭 智宅焕2 天前
智能家居全国交付中执行端能力评估与分级调度架构设计:从注册数量到交付确定性的系统实现
运维·智能家居·devops
吴建旭 智宅焕4 天前
智能家居全国交付中模块化计价架构的设计与实现:从一口价到可复核的计价模型
架构·智能家居·devops
7177779 天前
支持混合云的 DevOps 平台有哪些:Gitee、Azure DevOps、GitLab、Jenkins、JFrog 横向对比
gitee·azure·devops
小南家的青蛙11 天前
微服务体系下DevOps运维岗位面试方案设计
运维·面试·职场和发展·devops
逐流人13 天前
手工建集群到全自动发布:k8s集群部署与DevOps流水线整合实战
ci/cd·云原生·kubernetes·云计算·devops
无所畏俱的老尹16 天前
Agent 落地为什么需要影子对照:同输入只读双跑、写动作变提案、差异报警,比直接切全量更安全
人工智能·机器学习·云计算·aigc·devops
pengdott16 天前
Rocky linux9安装最新版sonarqube-26.10.0
ci/cd·devops·sonarqube
逐流人16 天前
DevOps 入门实战(一):从 CI/CD 概念到 Jenkins 自动构建部署
运维·云原生·云计算·jenkins·devops