Symptom-Based Alerting
Symptom-based alerting pages on user-visible problems rather than internal causes, reducing noise because many causes share one symptom.
Definition
Symptom-based alerting is the practice of triggering alerts on outcomes users experience, such as elevated errors or latency, instead of on every possible internal cause such as a single host's CPU.
Context and scope
It shapes on-call load and is central to keeping paging meaningful and actionable.
Common misconceptions
- alerting on every cause catches incidents sooner
- symptom alerts cannot be used to detect early degradation
Anti-patterns
- paging on a single host's metric without a user impact signal
- letting one outage fan out into dozens of correlated pages
Related concepts
Authoritative source
Definition
Symptom-based alerting is the practice of triggering alerts on outcomes users experience, such as elevated errors or latency, instead of on every possible internal cause such as a single host’s CPU.
It shapes on-call load and is central to keeping paging meaningful and actionable. Compared to simpler alternatives, this concept fits best when the trade-off it represents is the one the system actually cares about.
When to use it
A common use case is applying it at the boundary where the cost of getting it wrong is highest, and the scenario usually involves a concrete guarantee the system must uphold. Unlike an ad hoc approach, it makes that guarantee explicit, reviewable, and testable.
How it works
Because many internal causes produce the same user-visible symptom, alerting on the symptom keeps the page count low while still catching the incident.
Cause-based signals are still valuable as dashboards and diagnostics, but they should not page unless they map to a real impact.
Trade-offs and alternatives
A small number of high-quality symptom alerts, often tied to error budget burn, preserves trust in the pager instead of training responders to ignore it. The main drawback is the extra complexity it introduces, so the benefit should clearly outweigh that cost before a team adopts it.
Common misconceptions
A persistent misconception is that alerting on every cause catches incidents sooner. Another is that symptom alerts cannot be used to detect early degradation. For example, an engineer who assumes otherwise may design around a guarantee that does not actually exist, which only becomes obvious under load or failure.
Anti-patterns to avoid
Two recurring anti-patterns are to paging on a single host’s metric without a user impact signal and to letting one outage fan out into dozens of correlated pages. Both tend to surface as hard-to-debug production issues rather than obvious test failures. Related concepts include Trace Context Propagation, Exemplars Linking Metrics to Traces, Operational Dashboard Design.