Telemetry Pipeline Architecture
A telemetry pipeline architecture collects, processes, and routes telemetry from every source to every sink, so instrumentation is decoupled from the backend and a new sink does not require.
Definition
Telemetry pipeline architecture is the design of the system that receives telemetry from instrumented services — through agents, collectors, or direct export — processes it (sampling, enrichment, batching) and routes it to multiple sinks (tracing store, metrics store, log store), so that instrumentation is written once and the backend topology evolves independently.
Context and scope
Applied in OpenTelemetry Collector, observability platforms, and any organization with multiple telemetry backends or a backend that changes.
Common misconceptions
- The pipeline is just a queue — a queue is the transport; the pipeline also does processing (sampling, enrichment) and routing, which are the value
- A pipeline adds latency — it adds processing latency; it reduces network latency by batching and local egress, which usually nets out positive
Anti-patterns
- Sending every service directly to the backend, so changing the backend means reconfiguring every service
- Doing heavy processing in the pipeline that belongs in the instrumented service, creating a central bottleneck and a single point of failure
Related concepts
Authoritative source
What it is
Telemetry pipeline architecture is the design of the system that receives telemetry from instrumented services — through agents, collectors, or direct export — processes it (sampling, enrichment, batching) and routes it to multiple sinks (tracing store, metrics store, log store), so that instrumentation is written once and the backend topology evolves independently. Applied in OpenTelemetry Collector, observability platforms, and any organization with multiple telemetry backends or a backend that changes. The goal is not to collect all data but to be able to answer the question you did not think to ask in advance — which requires high-cardinality labels and exemplars that link traces to the metric that triggered them.
How it works
Three signals compose: metrics aggregate over time, traces preserve causality across boundaries, and logs carry the unstructured detail. Sampling is adaptive — head sampling for known-interesting paths, tail sampling to retain slow or error traces at a fixed budget. Cardinality is controlled at the source, not the query, because a high-cardinality label explodes storage and makes dashboards unusable.
When to invest and pitfalls
Invest when mean time to detect matters — in production, at scale, with paying users. The pitfall is instrumenting everything and alerting on nothing: dashboards full of lines that no one reads, alerts that fire on noise and are then muted, leaving the real outage undetected. Symptom-based alerts on user-visible degradation outperform cause-based alerts on internal thresholds.