Configure External Uptime Probes That Users Agree With

Probe from outside the datacenter on the real user path, alert on consecutive failures with downtime tracking, and page on the right severity.

~25 mindeployobservabilityoperations

Task

Set up external probes that detect outages users actually experience and alert with appropriate urgency and noise control.

When to use this

Internal health checks showed green while the CDN was down for users. The first alert came from Twitter, forty minutes into the incident.

Prerequisites

An external probing service or a probe runner outside your infrastructure
The exact user-facing URL sequence to monitor
An alerting channel with severity routing, such as PagerDuty

Steps

1. Probe the full user path, not the origin.

checks:
  - name: checkout-journey
    requests:
      - GET https://example.com/            # landing, via CDN
      - GET https://api.example.com/products  # API, via gateway
      - POST https://api.example.com/cart     # write path

Expected output: The probe exercises CDN, gateway, and API exactly as a user does, so any broken layer fails the check.

2. Alert on consecutive failures, not single blips.

alerting:
  failure_threshold: 3      # three consecutive failures
  interval: 30s
  severity: critical
  escalation: pagerduty

Expected output: One lost packet from the probe network does not page anyone; three failures in ninety seconds does.

3. Track downtime separately from incidents.

curl -s "https://probe.example.com/api/checks/checkout-journey/uptime?window=30d" | jq '.uptime_pct, .downtime_ms'

Expected output: The 30-day uptime percentage and total downtime are queryable facts for reviews, not estimates.

4. Add a region-aware probe for geographic blindness.

curl -s "https://probe.example.com/api/checks/checkout-journey/regions" | jq '.[] | {region, state}'

Expected output: Probes from multiple regions catch DNS or CDN failures that a single vantage point cannot see.

5. Verify the alert actually fires end to end.

probectl pause-origin 60 && sleep 120 && pdctl incidents list --since 5m

Expected output: A deliberate outage creates a real page within the expected threshold, proving the chain from failure to phone.

Troubleshooting

Problem

Probe fails but users are fine

Solution

The probe hits an internal-only endpoint or a bot-challenged path; probe the same URL a user types, and allow-list the probe user agent where challenges apply.

Problem

Alerts fire for every brief deploy

Solution

Failure threshold is too tight for your deploy drain time; align threshold times interval with your maximum safe maintenance duration.

Problem

Probe blocked by rate limiting

Solution

The probe looks like an attack; give probes a distinct header or user agent and exempt it from the limiter, keeping the path itself unmodified.

Limitations

External probes measure availability, not performance; latency monitoring needs a separate synthetic timing setup.
Write-path probes create test data; use a sandbox namespace or clean up probe artifacts on a schedule.

Green inside, broken outside

Internal health checks measure your servers; users measure the whole chain including DNS, CDN, and the TLS handshake. Only a probe outside your infrastructure walks the user path, and only consecutive-failure logic keeps it from crying wolf on single blips. The combination detects the outages users experience and pages a human with enough evidence to act immediately.

checks:
  - name: checkout-journey
    requests:
      - GET https://example.com/
      - GET https://api.example.com/products
      - POST https://api.example.com/cart

alerting:
  failure_threshold: 3
  interval: 30s
  severity: critical

Three failures, one page

Threshold tuning is noise control. One failure is a blip, three in ninety seconds is an outage, and the number should be derived from your deploy drain duration: if maintenance legally takes two minutes, a threshold that fires before that will page you for every scheduled change. Derive the threshold from reality, write it down, and revisit it when deploys change.

Downtime is an accounting, not a vibe

Uptime percentages quoted in reviews should come from the probe ledger, not from memory. The same records feed error budgets: minutes of downtime per month, measured on the user path, are the honest input to how much release velocity the reliability budget still allows.