Static Thresholds Versus Anomaly Detection
Static threshold alerts fire on fixed lines — latency over 500ms — while anomaly detection learns baselines and fires on deviations, each with distinct failure modes of noise and blindness.
Definition
Static alerts compare metrics against fixed thresholds chosen from experience. Anomaly detection models the metric normal behavior — seasonality, trend, variance — and alerts when observations deviate beyond learned bounds. Static thresholds are explainable and predictable but blind to gradual regressions and noisy under traffic seasonality; anomaly detection adapts but can learn to accept ongoing degradation and fails silently when baselines are wrong.
Context and scope
Chosen per metric and per on-call capacity. Error rates and SLOs lean static with burn-rate math; traffic-dependent and multi-seasonal metrics lean anomaly-based or hybrid.
Common misconceptions
- Anomaly detection replaces thresholds — it replaces thresholds where baselines are learnable; contracts and SLOs want fixed lines with business meaning
- Anomalies mean incidents — anomalies mean deviations; many are deploys, campaigns, and calendar events, and the alert design must anticipate that
- Alert fatigue is a tooling problem — it is a design problem: every alert must be actionable, and tuning is the ongoing cost of either approach
Anti-patterns
- Replacing every static alert with anomaly detection and drowning on-call in deploy-time deviations
- Training baselines on incident-contaminated history so the model learns to accept the outage as normal
- Setting static thresholds from one week of data and paging on every weekend traffic dip
Related concepts
Authoritative source
What it is
Two theories of “when to page”. Fixed lines: humans decide what bad means and the alert enforces it. Learned baselines: history decides what normal means and the alert enforces deviation. Each fails differently: fixed lines blind and noisy in the wrong places; learned lines drifting and opaque.
How it works
Static alerts need threshold selection from percentile history and traffic awareness — the same absolute error count is an incident at night and noise at noon. Anomaly models fit seasonality — hourly, daily, weekly — and flag residual deviations beyond robust variance. Deployment markers let humans separate expected deviations from real ones.
When to choose which
SLO error budgets and contractual limits: static, with burn-rate windows. Traffic-scaled internals — queue depths, cache hit ratios, per-region latencies: anomaly-based or ratio-normalized static. The hybrid rule: page on fixed lines with business meaning, ticket on anomalies worth human review.
Pitfalls
Anomaly detection that includes bad history normalizes degradation — train and update windows need incident exclusion. Long gradual regressions defeat both approaches without week-over-week comparisons. Every anomaly alert needs an explanation attached — the model confidence and the deviation window — or triage becomes archaeology.