Observability-Driven Testing
Observability-driven testing uses traces, metrics, and logs as test evidence, asserting on system behavior rather than only on returned values.
Definition
Observability-driven testing designs and verifies tests through telemetry, using traces, metrics, and logs to confirm that internal behavior matches expectations.
Context and scope
Used for distributed services where a correct response can hide retries, slow dependencies, missing spans, or silent fallbacks.
Common misconceptions
- Observability-driven testing means watching dashboards while tests run
- Passing functional assertions makes telemetry assertions unnecessary
- It applies only in production, never in automated tests
Anti-patterns
- Asserting on log strings that change with every refactor
- Collecting telemetry during tests but never asserting on it
- Testing observability by checking that a dashboard loads
Related concepts
Related examples
Authoritative source
https://opentelemetry.io/docs/concepts/observability-primer/
Beyond the response body
A request can return 200 while doing the wrong thing internally: retrying three times, falling back to stale cache, skipping an audit event, or calling a deprecated endpoint. Functional assertions see only the response. Observability-driven testing treats traces, metrics, and logs as first-class test evidence. The test asserts that the expected spans exist, that retry counts are zero, that the cache was consulted, and that latency stayed within budget.
What to assert
Assert on stable semantics: span names, parent-child relationships, status codes, and documented attributes such as HTTP route, database operation, or messaging destination. Assert that sensitive fields are absent. Assert that a failed dependency produced the expected fallback rather than an unhandled error. Metric assertions can check counters and histograms, but they need deterministic scopes so parallel tests do not interfere.
Instrumentation as part of the feature
Telemetry is code. Treat span attributes and metric labels as part of the interface under test, and review them like API contracts. Missing spans or high-cardinality labels are defects, not cosmetic issues. A feature that cannot be observed is hard to operate, so testing its telemetry is part of testing the feature. Compared to testing only outputs, this approach catches silent degradation earlier.
Test environments
Run these assertions in integration and end-to-end tests with an in-memory or local exporter. Avoid depending on a shared observability backend, which makes tests slow and flaky. For example, a test can capture spans in memory and assert that a downstream timeout produced exactly one retry and one fallback span.
Limits
Telemetry assertions are brittle if they depend on free-text logs or implementation details. They also add setup cost and can slow suites. Use them where behavior is otherwise invisible: retries, fallbacks, async work, and cross-service flows. Pair them with dashboards and synthetic checks so the same signals are used in production.