Escalate Unacknowledged Alerts Automatically

Configure Alertmanager grouping and repeat intervals plus a two-tier escalation path, so a missed page reaches someone awake.

~30 minalertingoncallreliability

Task

Make sure a critical alert is acknowledged, and escalate to a second responder when it is not.

When to use this

A critical database alert fires at 02:00. The primary on-call phone is on silent, and the first human sees the message at 09:00 in a muted channel.

Prerequisites

Prometheus with Alertmanager
An on-call provider such as PagerDuty or Opsgenie
Webhook credentials for the paging receiver

Steps

1. Route critical alerts to the pager with a short repeat interval and keep everything else in chat.

route:
  receiver: chat
  group_by: [alertname, service]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity="critical"
      receiver: pager
      repeat_interval: 15m
receivers:
  - name: chat
    webhook_configs:
      - url: https://chat.example.com/hooks/alerts
  - name: pager
    webhook_configs:
      - url: https://events.pagerduty.com/v2/enqueue
        send_resolved: true

Expected output: Critical alerts page immediately and re-notify every 15 minutes until they resolve.

2. Configure the provider escalation policy: primary at once, secondary after five minutes, manager after fifteen.

Expected output: An unacknowledged page reaches the secondary on-call after five minutes and the manager after fifteen.

3. Use scoped silences for maintenance instead of muting the channel.

amtool silence add alertname=~"DiskPressure.*" --duration=2h --comment="node pool upgrade"

Expected output: The silence is scoped, attributed to a user, has an expiry, and appears in the Alertmanager UI.

4. Review alert timing after every incident to catch pages nobody saw.

amtool alert query --alertmanager.url=http://alertmanager:9093

Expected output: Alert counts and timings show whether the first notification was acknowledged and which alerts flap.

Troubleshooting

Problem

Alerts from different services arrive as one page

Solution

Group by service and alertname rather than by cluster; over-grouping hides the actionable alert inside a bundle.

Problem

Escalation reaches the wrong person

Solution

Provider schedules and Alertmanager routing must agree. Verify the service mapping and any calendar overrides.

Limitations

Escalation cannot fix a broken paging integration; monitor the alerting path with a dead man's switch.
Aggressive repeat intervals cause alert fatigue, which leads to muting and defeats escalation.

An alert nobody sees is not an alert

Paging systems assume the first responder will acknowledge, and escalation exists for the case where they do not. That assumption fails quietly: a phone on silent, a webhook pointed at a decommissioned integration, or a notification buried in a busy channel all look identical from the monitoring side. The fix is a two-tier policy with a measurable timer. The primary gets the page, the secondary gets it after five minutes without acknowledgement, and a manager gets it after fifteen. Every tier should be tested, not just configured.

Route by severity, group by service

Alertmanager decides who receives what before the provider sees it. Route critical alerts to the paging receiver and everything else to chat, and group by service and alert name so a single incident does not produce twenty pages. Keep repeat_interval short enough for critical alerts that a missed page comes back, but not so short that the same unresolved issue pages every two minutes. Those two settings, more than any dashboard, determine whether on-call is sustainable.

routes:
  - matchers:
      - severity="critical"
    receiver: pager
    repeat_interval: 15m

Silence with a scope and an owner

A muted channel is an invisible outage of the alerting system. Use amtool silence add with a matcher, a duration, and a comment naming the change or incident, so the silence expires on its own and the next responder can see why it existed. Review silences weekly; a silence that keeps getting extended is a signal that the alert is wrong, not that the maintenance is long. Finally, add a dead man’s switch that alerts when no alerts have arrived for a day, because that is the only reliable way to detect a broken pipeline.