08 Inside the Collector: Pipelines, Deployment Patterns, and Failure Modes

Inside the Collector: Pipelines, Deployment Patterns, and Failure Modes

Part 8 of a series on observability for microservices. Part 7 covered signals and context propagation. This post covers the piece that moves telemetry once it leaves your process: the OpenTelemetry Collector. Series index.

📦 GitHub: https://github.com/geekchow/O11y-Micro-Service

What it is, in one line

The Collector is a single Go binary that runs configurable receivers → processors → exporters pipelines --- the vendor-agnostic middle tier that lets apps offload telemetry fast and lets backends become a config choice instead of a code change.

Could your app export straight to a backend, skipping the Collector entirely? Yes --- for local dev or a tiny setup, that's fine. Production wants the middle tier because it centralizes everything you must not scatter across fifty services: retries, batching, credentials, PII scrubbing, sampling policy, and vendor routing.

The five component types

Type	Role	Workhorses you'll actually use
Receivers	Data in (listen or scrape)	`otlp` (gRPC `:4317` / HTTP `:4318`); `prometheus` (scrapes targets!), `filelog` (tails files), `jaeger`, `kafka`, `hostmetrics`
Processors	Transform in flight	`memory_limiter` (always first), `batch` (always ~last), `attributes`/`resource` (add/drop/rename), `filter`, `transform` (OTTL language), `k8sattributes` (pod metadata enrichment), `tail_sampling`
Exporters	Data out, with retry + persistent queue	`otlp`/`otlphttp`, `prometheusremotewrite`, `debug` (to console), vendor exporters (`splunk_hec`, `datadog`...)
Connectors	Exporter of pipeline A and receiver of pipeline B	`spanmetrics` (spans → RED metrics), `servicegraph`, `forward`
Extensions	Side services, touch no telemetry	`health_check`, `pprof`, `zpages` (live pipeline debugging)

Config anatomy: define, then wire

The config has two halves, and the second half is the one people forget: defining a component does nothing until a pipeline references it.

yaml 复制代码

receivers:
  otlp:
    protocols: { grpc: {}, http: {} }
processors:
  memory_limiter: { check_interval: 1s, limit_percentage: 80 }
  batch: {}
exporters:
  otlphttp/tempo: { endpoint: http://tempo:4318 }
  prometheusremotewrite: { endpoint: http://mimir:9009/api/v1/push }
connectors:
  spanmetrics: {}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/tempo, spanmetrics]   # fan-out: both get every batch
    metrics:
      receivers: [otlp, spanmetrics]             # ← connector re-enters here
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]

#mermaid-svg-MJ3xpqjJ9ecar9r8{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-MJ3xpqjJ9ecar9r8 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .error-icon{fill:#552222;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .marker.cross{stroke:#333333;}#mermaid-svg-MJ3xpqjJ9ecar9r8 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-MJ3xpqjJ9ecar9r8 p{margin:0;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .cluster-label text{fill:#333;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .cluster-label span{color:#333;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .cluster-label span p{background-color:transparent;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .label text,#mermaid-svg-MJ3xpqjJ9ecar9r8 span{fill:#333;color:#333;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .node rect,#mermaid-svg-MJ3xpqjJ9ecar9r8 .node circle,#mermaid-svg-MJ3xpqjJ9ecar9r8 .node ellipse,#mermaid-svg-MJ3xpqjJ9ecar9r8 .node polygon,#mermaid-svg-MJ3xpqjJ9ecar9r8 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .rough-node .label text,#mermaid-svg-MJ3xpqjJ9ecar9r8 .node .label text,#mermaid-svg-MJ3xpqjJ9ecar9r8 .image-shape .label,#mermaid-svg-MJ3xpqjJ9ecar9r8 .icon-shape .label{text-anchor:middle;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .rough-node .label,#mermaid-svg-MJ3xpqjJ9ecar9r8 .node .label,#mermaid-svg-MJ3xpqjJ9ecar9r8 .image-shape .label,#mermaid-svg-MJ3xpqjJ9ecar9r8 .icon-shape .label{text-align:center;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .node.clickable{cursor:pointer;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .arrowheadPath{fill:#333333;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-MJ3xpqjJ9ecar9r8 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-MJ3xpqjJ9ecar9r8 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-MJ3xpqjJ9ecar9r8 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .cluster text{fill:#333;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .cluster span{color:#333;}#mermaid-svg-MJ3xpqjJ9ecar9r8 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-MJ3xpqjJ9ecar9r8 rect.text{fill:none;stroke-width:0;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .icon-shape,#mermaid-svg-MJ3xpqjJ9ecar9r8 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .icon-shape p,#mermaid-svg-MJ3xpqjJ9ecar9r8 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .icon-shape .label rect,#mermaid-svg-MJ3xpqjJ9ecar9r8 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-MJ3xpqjJ9ecar9r8 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-MJ3xpqjJ9ecar9r8 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-MJ3xpqjJ9ecar9r8 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} metrics pipeline
traces pipeline
otlp receiver
memory_limiter
batch
otlphttp → Tempo
spanmetrics connector
memory_limiter
batch
remote_write → Mimir
otlp receiver

Three rules of thumb worth internalizing: memory_limiter goes first (refuse before you swell), batch goes last before exporters (compress the network), and one pipeline per signal type --- with the name/instance syntax (otlphttp/tempo) when you need multiple instances of one component type, exactly as the config above does.

On distributions : the same engine ships in flavors --- core (minimal, curated), contrib (~everything; this is where tail_sampling, k8sattributes, and vendor exporters live, and the usual choice), k8s , and otlp-only . For production hardening, build a custom binary containing only the components you actually use with the OpenTelemetry Collector Builder (ocb). Component maturity varies individually per component --- check each one's README, not just the distribution as a whole.

A real gateway config, annotated

Here's the tail-sampling gateway Collector from the companion stack, with the parts that trip people up called out:

yaml 复制代码

connectors:
  spanmetrics:
    namespace: traces.span.metrics
    exemplars:
      enabled: true

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 256
    spike_limit_mib: 64
  batch: {}
  tail_sampling:
    decision_wait: 10s
    num_traces: 20000
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow
        type: latency
        latency: { threshold_ms: 2000 }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 25 }

service:
  pipelines:
    traces:                       # the culled stream → Tempo
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp/tempo]
    traces/spanmetrics:           # the UNCUT stream → RED metrics
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [spanmetrics]

That second pipeline --- traces/spanmetrics --- is not an accident or duplication. It reads the same incoming OTLP spans as the traces pipeline but skips tail_sampling entirely, because RED metrics need to see every request, not just the ones tail sampling decided to keep. Get this ordering backwards --- run spanmetrics after sampling --- and your dashboard's error rate silently understates reality by whatever fraction tail sampling discarded.

Deployment patterns --- the decision that actually matters

Pattern	Topology	Choose when
No collector	SDK → backend directly	Dev, demos, tiny setups
Agent	SDK → Collector on the same node (DaemonSet/sidecar) → backend	Default production baseline: apps offload in microseconds, agent adds node/pod metadata no central box could know
Gateway	SDKs/agents → load-balanced central Collector tier → backends	You need fleet-wide policy in one place: tail sampling, PII scrubbing, egress credentials, per-vendor routing
Agent + gateway	Both	The common end-state at scale

The trade is locality vs centrality: agents know node things and sit near the app; gateways see whole traces and hold one copy of policy and secrets. Tail sampling forces a gateway tier by definition --- a per-node agent can never see all the spans of a distributed trace, since those spans originate on different nodes. Part 9 covers the routing trick this requires when you scale the gateway tier past a single instance.

Failure modes --- how the pipeline sheds load

The Collector pipeline is a chain of queues, and it's worth knowing exactly how each link fails:
#mermaid-svg-9hYUciI7DCpuqMPX{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-9hYUciI7DCpuqMPX .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-9hYUciI7DCpuqMPX .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-9hYUciI7DCpuqMPX .error-icon{fill:#552222;}#mermaid-svg-9hYUciI7DCpuqMPX .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-9hYUciI7DCpuqMPX .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-9hYUciI7DCpuqMPX .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-9hYUciI7DCpuqMPX .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-9hYUciI7DCpuqMPX .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-9hYUciI7DCpuqMPX .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-9hYUciI7DCpuqMPX .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-9hYUciI7DCpuqMPX .marker{fill:#333333;stroke:#333333;}#mermaid-svg-9hYUciI7DCpuqMPX .marker.cross{stroke:#333333;}#mermaid-svg-9hYUciI7DCpuqMPX svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-9hYUciI7DCpuqMPX p{margin:0;}#mermaid-svg-9hYUciI7DCpuqMPX .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-9hYUciI7DCpuqMPX .cluster-label text{fill:#333;}#mermaid-svg-9hYUciI7DCpuqMPX .cluster-label span{color:#333;}#mermaid-svg-9hYUciI7DCpuqMPX .cluster-label span p{background-color:transparent;}#mermaid-svg-9hYUciI7DCpuqMPX .label text,#mermaid-svg-9hYUciI7DCpuqMPX span{fill:#333;color:#333;}#mermaid-svg-9hYUciI7DCpuqMPX .node rect,#mermaid-svg-9hYUciI7DCpuqMPX .node circle,#mermaid-svg-9hYUciI7DCpuqMPX .node ellipse,#mermaid-svg-9hYUciI7DCpuqMPX .node polygon,#mermaid-svg-9hYUciI7DCpuqMPX .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-9hYUciI7DCpuqMPX .rough-node .label text,#mermaid-svg-9hYUciI7DCpuqMPX .node .label text,#mermaid-svg-9hYUciI7DCpuqMPX .image-shape .label,#mermaid-svg-9hYUciI7DCpuqMPX .icon-shape .label{text-anchor:middle;}#mermaid-svg-9hYUciI7DCpuqMPX .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-9hYUciI7DCpuqMPX .rough-node .label,#mermaid-svg-9hYUciI7DCpuqMPX .node .label,#mermaid-svg-9hYUciI7DCpuqMPX .image-shape .label,#mermaid-svg-9hYUciI7DCpuqMPX .icon-shape .label{text-align:center;}#mermaid-svg-9hYUciI7DCpuqMPX .node.clickable{cursor:pointer;}#mermaid-svg-9hYUciI7DCpuqMPX .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-9hYUciI7DCpuqMPX .arrowheadPath{fill:#333333;}#mermaid-svg-9hYUciI7DCpuqMPX .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-9hYUciI7DCpuqMPX .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-9hYUciI7DCpuqMPX .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-9hYUciI7DCpuqMPX .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-9hYUciI7DCpuqMPX .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-9hYUciI7DCpuqMPX .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-9hYUciI7DCpuqMPX .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-9hYUciI7DCpuqMPX .cluster text{fill:#333;}#mermaid-svg-9hYUciI7DCpuqMPX .cluster span{color:#333;}#mermaid-svg-9hYUciI7DCpuqMPX div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-9hYUciI7DCpuqMPX .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-9hYUciI7DCpuqMPX rect.text{fill:none;stroke-width:0;}#mermaid-svg-9hYUciI7DCpuqMPX .icon-shape,#mermaid-svg-9hYUciI7DCpuqMPX .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-9hYUciI7DCpuqMPX .icon-shape p,#mermaid-svg-9hYUciI7DCpuqMPX .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-9hYUciI7DCpuqMPX .icon-shape .label rect,#mermaid-svg-9hYUciI7DCpuqMPX .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-9hYUciI7DCpuqMPX .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-9hYUciI7DCpuqMPX .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-9hYUciI7DCpuqMPX :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} )|OTLP| RCV $receiver$

RCV
backpressure
queue full → drop

this exporter only
refusal propagates as

OTLP error to SDK
SDK BatchSpanProcessor

bounded queue → drops
memory_limiter

refuses batches under pressure
processors
exporter queue

retry w/ backoff

optional file_storage persistence
backend

down or slow
dropped

Backend down: that exporter's queue absorbs the burst, retries with backoff, and eventually drops --- but other exporters in the same fan-out keep flowing normally. This is per-backend blast-radius isolation. Add the file_storage extension for a queue that survives a Collector restart.
Collector overwhelmed: memory_limiter starts refusing incoming batches; the refusal propagates back as an OTLP error to the SDK; the SDK's own queue fills and drops at the source. Every stage degrades gracefully rather than blocking the application thread or OOM-killing the Collector.
Collector down entirely: the SDK retries briefly, then drops. This is exactly why the agent tier runs on-node --- there's almost nothing to lose on that first hop --- and why the Collector's own health_check and internal metrics feed back into Prometheus. It's the "watcher's watcher" pattern applied to the observability pipeline itself.

Given any Collector YAML, you should now be able to draw its pipelines on a whiteboard, argue agent-vs-gateway for a given org's needs, and, for any failure scenario ("Splunk ingest is lagging"), say exactly which queue fills and what gets dropped.

Next: the policy that the gateway tier exists to run --- sampling.

➡️ Next: Part 9 --- Sampling: Keeping the Interesting 1%