Observability Gap Analysis
Observability gap analysis walks the incident lifecycle and finds the steps where the operator was blind — no trace, no log, no metric — so the instrumentation backlog is prioritized by diagnosis.
Definition
Observability gap analysis is the practice of reviewing past incidents and mapping each diagnosis step to the telemetry that supported it, so that the steps where the operator was forced to guess — no trace across a hop, no log on the failing path — become the prioritized instrumentation backlog.
Context and scope
Applied in platform teams, SRE programs, and any organization that treats instrumentation as a product with a backlog rather than a one-time setup.
Common misconceptions
- Gap analysis is just adding more logging — it is adding the right telemetry at the gaps that blocked diagnosis, not more volume everywhere
- If we have dashboards we have observability — dashboards show aggregates; gaps are in the instance-level traces and logs that explain a specific failure
Anti-patterns
- Instrumenting every code path equally, paying storage and overhead for telemetry that is never used in diagnosis
- Doing gap analysis once and never again, so new gaps from new services accumulate unmeasured
Related concepts
Authoritative source
https://charity.wtf/2017/03/09/science-is-about-proving-yourself-wrong/
What it is
Observability gap analysis is the practice of reviewing past incidents and mapping each diagnosis step to the telemetry that supported it, so that the steps where the operator was forced to guess — no trace across a hop, no log on the failing path — become the prioritized instrumentation backlog. Applied in platform teams, SRE programs, and any organization that treats instrumentation as a product with a backlog rather than a one-time setup. The goal is not to collect all data but to be able to answer the question you did not think to ask in advance — which requires high-cardinality labels and exemplars that link traces to the metric that triggered them.
How it works
Three signals compose: metrics aggregate over time, traces preserve causality across boundaries, and logs carry the unstructured detail. Sampling is adaptive — head sampling for known-interesting paths, tail sampling to retain slow or error traces at a fixed budget. Cardinality is controlled at the source, not the query, because a high-cardinality label explodes storage and makes dashboards unusable.
When to invest and pitfalls
Invest when mean time to detect matters — in production, at scale, with paying users. The pitfall is instrumenting everything and alerting on nothing: dashboards full of lines that no one reads, alerts that fire on noise and are then muted, leaving the real outage undetected. Symptom-based alerts on user-visible degradation outperform cause-based alerts on internal thresholds.