Distributed Tracing
Distributed tracing follows one request across services with a shared trace id and timed spans, revealing where latency and failures originate.
Definition
Distributed tracing records a request as a trace made of spans. Each span represents a unit of work with a start time, duration, attributes, and a parent-child relationship. A trace id is propagated across service boundaries so spans from different services can be assembled into one timeline. Traces reveal latency, errors, and call paths that metrics and logs cannot show alone.
Context and scope
Tracing is used in microservices, serverless, and any request path that crosses process boundaries. It is especially valuable when a single user action fans out to many services and the slow step is not obvious.
Common misconceptions
- Tracing replaces metrics or logs — each answers a different question and all three are complementary
- Every request must be traced — sampling controls cost; tail-based sampling keeps the interesting traces
- A trace id in logs is enough — without spans and context propagation, it does not show where time was spent
Anti-patterns
- Generating a new trace id at each service instead of propagating the incoming one
- Adding high-cardinality user data to span attributes
- Sampling only successful requests and losing the failures
Related concepts
Related examples
Authoritative source
A trace is a timeline
A request enters the system and receives a trace id. Each service creates a span, records its work, and passes the trace context to downstream calls. The backend assembles spans into a tree. The slowest span and the error span show where to investigate.
trace 4bf92f...
gateway 120 ms
orders 95 ms
postgres 70 ms
payments 20 msThe W3C traceparent header carries the trace id, parent span id, and sampling decision. Propagate it through HTTP clients, message headers, and background jobs. A service that starts a new trace at every hop breaks the timeline.
Sampling and cost
Tracing every request is expensive at high volume. Head-based sampling decides at the start and is simple but may miss rare failures. Tail-based sampling buffers spans and keeps traces that are slow or contain errors, at the cost of infrastructure. Regardless of method, record the sampling decision in the context so all services agree.
Instrument boundaries, not every line
Instrument inbound requests, outbound calls, database queries, and queue operations. Add attributes that help diagnosis, such as route, status, and dependency name, but avoid secrets and unbounded user data. Link traces to logs with the trace id and to metrics with exemplars where supported.
When to use it
Use distributed tracing when a request crosses multiple services and latency or failure attribution is unclear. The advantage is end-to-end visibility and precise blame; the drawback is instrumentation, storage cost, and sampling complexity. Compared to metrics, traces explain individual requests; compared to logs, they show the call path and timing without reading thousands of lines.