Alert Fatigue
Alert fatigue is the desensitization that follows too many low-quality alerts, causing responders to miss or delay action on the signals that matter.
Definition
Alert fatigue is a state in which responders become numb to alerts because of volume, noise, or poor routing, reducing their ability to react to genuine incidents.
Context and scope
Common in teams with threshold alerts on every metric, shared notification channels, and no ownership of alert quality.
Common misconceptions
- More alerts mean better coverage
- Fatigue is a people problem rather than a system design problem
- Muting noisy alerts is a sufficient solution
Anti-patterns
- Alerting on every error log line
- Sending the same notification to several channels with no owner
- Keeping alerts after their runbook or service is retired
Related concepts
Authoritative source
What alert fatigue is
Alert fatigue is the gradual loss of response that follows repeated exposure to alerts that do not require action. Responders learn that most pages are noise, so they acknowledge without investigating, delay response, or mute channels. The cost is invisible until a real incident waits in a queue behind routine notifications. Fatigue is a property of the alerting system, not a character flaw in the responder.
How it accumulates
Common causes include alerting on causes rather than symptoms, setting thresholds without baselines, and routing every service’s alerts to one channel. Duplicate alerts from multiple tools inflate volume. Alerts with no runbook force investigation from scratch. Ownership gaps mean nobody removes obsolete checks. During incidents, a storm of related alerts hides the initiating failure. The result is a system that reports constantly and communicates poorly.
Symptom-based, SLO-driven alerting
Page on what users feel: error rate, latency, and saturation relative to an objective. Let dashboards and tickets carry the detailed causes. Use multi-window burn rates so a brief blip does not page, while sustained degradation does. Group related alerts into one incident. Route by service ownership, and give every page a runbook link and a clear action. Compared to alerting on every CPU spike, this approach pages less and means more.
Hygiene
Review alert volume and action rate monthly. For each rule ask: did it fire, did anyone act, and would a user have noticed? Delete or downgrade rules that never lead to action. Deduplicate across tools. After an incident, check whether the alerting helped or hindered, and record the finding. Set a paging budget per rotation and treat overruns as a defect.
Culture
For example, a team that pages on every failed health check will soon ignore health checks. A team that reviews noisy alerts and fixes them builds trust that a page always deserves attention. Leadership must protect the time to do this work, or fatigue returns.