中文
devexamples
← Workflows

阅读中文版本

Alert Noise Reduction Workflow

Measure which alerts fire most and how many are actionable, suppress or fix the noisy ones, and reserve paging for conditions that need a human now.

intermediate~90 minobservabilityoncalloperations

Objective

Cut paging noise so every page is actionable and on-call attention goes to real problems.

Environment

  • Alerting data on volume and actionability
  • The alerting configuration

Steps

1. Rank alerts by frequency

Rank alerts by frequency and by whether they were actionable.

Expected state: The noisiest and least actionable are identified.

2. Fix root causes or lower

Fix root causes or lower the severity of non-actionable alerts.

Expected state: Noise is removed at the source.

3. Route informational alerts away from paging

Route informational alerts away from paging.

Expected state: Only real problems page.

4. Review after changes to confirm paging

Review after changes to confirm paging volume dropped.

Expected state: The reduction is verified.

Decision points

Suppress or fix the alert?

  • Fix the source
  • Suppress

Fix the source where possible; suppression hides a real signal that may matter later.

Page on symptoms or causes?

  • Symptoms users feel
  • Causes

Page on user-impacting symptoms; causes are for dashboards unless they always lead to impact.

Expected outcome

A paging rotation where alerts are actionable, noise is reduced at the source, and informational signals do not page.

Verification

Confirm paging volume dropped and that each remaining page was actionable in the review period.

Cleanup

Record which alerts were changed and why, and update the runbook links.

Overview

Cut paging noise so every page is actionable and on-call attention goes to real problems. Confirm paging volume dropped and that each remaining page was actionable in the review period.

Steps

  1. Rank alerts by frequency and by whether they were actionable. Expected state: The noisiest and least actionable are identified.
  2. Fix root causes or lower the severity of non-actionable alerts. Expected state: Noise is removed at the source.
  3. Route informational alerts away from paging. Expected state: Only real problems page.
  4. Review after changes to confirm paging volume dropped. Expected state: The reduction is verified.
# top pagers by count
awk '{print $3}' alerts.log | sort | uniq -c | sort -nr | head
# then review actionability for each

Verification

  • A paging rotation where alerts are actionable, noise is reduced at the source, and informational signals do not page.

Decision points

  • Suppress or fix the alert? Options: Fix the source; Suppress. Recommended: Fix the source where possible; suppression hides a real signal that may matter later.
  • Page on symptoms or causes? Options: Symptoms users feel; Causes. Recommended: Page on user-impacting symptoms; causes are for dashboards unless they always lead to impact.