Fail Over DNS to a Standby Region

Create health-checked primary and secondary records, verify the failover with a forced health check, and account for TTL lag.

~35 mindnsfailoverreliability

Task

Route traffic to a standby region automatically when the primary endpoint stops answering health checks.

When to use this

The primary region loses its load balancer. DNS still returns the dead address for the full TTL, so users see timeouts even though the standby is healthy.

Prerequisites

A Route 53 hosted zone, or a DNS provider with health-checked failover
A healthy standby deployment in a second region
Permission to create health checks and change records

Steps

1. Create a health check for the primary endpoint.

aws route53 create-health-check --caller-reference payments-primary-$(date +%s) --health-check-config Type=HTTPS,ResourcePath=/healthz,FullyQualifiedDomainName=primary.example.com,RequestInterval=30,FailureThreshold=3

Expected output: The health check is created, returns Healthy, and prints its ID for the records.

2. Create primary and secondary failover records that reference the health check.

{
  "Changes": [
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "api.example.com",
        "Type": "A",
        "SetIdentifier": "primary",
        "Failover": "PRIMARY",
        "HealthCheckId": "a1b2c3d4-5678-90ab-cdef-1234567890ab",
        "AliasTarget": {
          "HostedZoneId": "Z1D633PJN98FT9",
          "DNSName": "primary-lb.eu-west-1.elb.amazonaws.com",
          "EvaluateTargetHealth": true
        }
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "api.example.com",
        "Type": "A",
        "SetIdentifier": "secondary",
        "Failover": "SECONDARY",
        "AliasTarget": {
          "HostedZoneId": "Z1D633PJN98FT9",
          "DNSName": "secondary-lb.eu-west-1.elb.amazonaws.com",
          "EvaluateTargetHealth": true
        }
      }
    }
  ]
}

Expected output: Queries resolve to the primary while it is healthy and to the secondary after three failed checks.

3. Test the failover without breaking production by disabling the health check briefly.

aws route53 update-health-check --health-check-id a1b2c3d4-5678-90ab-cdef-1234567890ab --disabled && sleep 120 && dig +short api.example.com

Expected output: The answer changes to the standby address within about two minutes; re-enable the check afterwards.

4. Confirm the TTL and that resolvers observe the change.

dig api.example.com | grep -A 2 'ANSWER SECTION'

Expected output: The record's TTL is short enough for the required recovery time, and the answer is the secondary address.

Troubleshooting

Problem

Failover never triggers

Solution

The health check must be associated with the record and enabled. Check the health check status history for the failing path.

Problem

Users still hit the failed region after the switch

Solution

Cached DNS and existing TCP connections keep old addresses. Lower the TTL before planned failovers and drain connections at the edge.

Limitations

DNS failover is slow by design and cannot meet a sub-minute recovery time for existing connections.
A single probe path can report failure while most of the application is still healthy.

DNS failover is a last resort with a delay

Health-checked DNS moves traffic between endpoints without a load balancer in front, which makes it useful for regional failover and simple to reason about. It is also slow: the health check needs several failures before it marks an endpoint down, resolvers cache the previous answer for the TTL, and already-open connections ignore DNS entirely. Plan for minutes, not seconds, and make the standby capable of absorbing full traffic before you rely on it. The record change must be tested, because an untested failover usually fails at the worst moment.

Build the pair, then break it on purpose

Create a health check for the primary, attach it to the primary record, and add a secondary record without a health check. Use alias records for AWS load balancers so the health check evaluates the target rather than a DNS name that may resolve differently for the checker. Then disable the health check for a couple of minutes and watch a resolver return the secondary address. Re-enable it and confirm the primary comes back, because a failover that cannot fail back is a one-way migration.

aws route53 update-health-check --health-check-id a1b2c3d4-5678-90ab-cdef-1234567890ab --disabled
dig +short api.example.com

Prepare the rest of the path

DNS is only the first hop; the standby needs warm caches, the same secrets, and a database that can accept writes or a documented read-only mode. Keep the TTL low enough for your recovery objective but high enough to avoid resolver load, and lower it in advance of planned maintenance. Monitor the health check status itself, because a misconfigured check that always reports healthy is worse than no failover: it creates confidence without protection.