Alert Routing Design
Alert routing design directs each alert to the on-call team that owns the affected service, so the alert reaches the team that can fix it rather than a default queue where it is ignored.
Definition
Alert routing design is the practice of mapping each alert rule to the owning team through service ownership metadata and routing rules, so that a database alert goes to the database team and a frontend alert to the frontend team, minimizing the mean time to acknowledge by ensuring the alert reaches someone who can act.
Context and scope
Applied in PagerDuty, Opsgenie, and any on-call system where multiple teams share rotation and misrouted alerts cost minutes per incident.
Common misconceptions
- Routing by service name is sufficient — services are owned by teams, but ownership changes; routing must follow the ownership source of truth, not a static map
- Everyone-alerts ensure coverage — they ensure noise; every team gets every alert and the relevant one is lost in the flood
Anti-patterns
- Routing all alerts to a single global rotation, so every team pages for every service and ownership is rediscovered at 3 a.m.
- Routing by a label that the alerting rule does not always set, so some alerts fall to the default queue and are never seen
Related concepts
Authoritative source
What it is
Alert routing design is the practice of mapping each alert rule to the owning team through service ownership metadata and routing rules, so that a database alert goes to the database team and a frontend alert to the frontend team, minimizing the mean time to acknowledge by ensuring the alert reaches someone who can act. Applied in PagerDuty, Opsgenie, and any on-call system where multiple teams share rotation and misrouted alerts cost minutes per incident. The goal is not to collect all data but to be able to answer the question you did not think to ask in advance — which requires high-cardinality labels and exemplars that link traces to the metric that triggered them.
How it works
Three signals compose: metrics aggregate over time, traces preserve causality across boundaries, and logs carry the unstructured detail. Sampling is adaptive — head sampling for known-interesting paths, tail sampling to retain slow or error traces at a fixed budget. Cardinality is controlled at the source, not the query, because a high-cardinality label explodes storage and makes dashboards unusable.
When to invest and pitfalls
Invest when mean time to detect matters — in production, at scale, with paying users. The pitfall is instrumenting everything and alerting on nothing: dashboards full of lines that no one reads, alerts that fire on noise and are then muted, leaving the real outage undetected. Symptom-based alerts on user-visible degradation outperform cause-based alerts on internal thresholds.