Run a Multi-Region Failover Drill
Simulate loss of a primary region, shift traffic to a secondary, verify data freshness and writes, then fail back without leaving split-brain risk.
Objective
Measure the real recovery time and data loss window when a region fails, and prove that traffic, data, and operators can move together.
Environment
- A global load balancer or DNS layer with health checks
- A replicated database with a documented promotion procedure
- Region-specific application deployments and secrets
- A maintenance window and stakeholder communication plan
Steps
1. Record the baseline
Capture the active region, replication lag, request rate, error rate, and a checksum of a representative table.
Expected state: The baseline is stored and the secondary is within the allowed replication lag.
2. Simulate the region loss
Remove the primary region from the load balancer or block its health check, without deleting data. Stop writes only if the data plan requires it.
Expected state: The failure is visible to the control plane and the primary is no longer receiving new traffic.
3. Promote and shift traffic
Promote the secondary database, update application connection strings, and shift traffic according to the runbook.
Expected state: The secondary serves reads and writes and the global endpoint routes to it.
4. Verify application behavior
Run a write, read it back, test authentication, and check asynchronous jobs and caches against the new region.
Expected state: Critical user flows work end to end, not only the health endpoint.
5. Measure and fail back
Record detection, promotion, traffic shift, and verification durations. Restore replication toward the original region and fail back in a controlled step.
Expected state: RTO/RPO are measured, and the original topology is restored without divergent writes.
Decision points
Should the drill stop writes during promotion?
- Yes, to guarantee a clean RPO
- No, to preserve availability and accept a small loss window
Stop writes when the data loss tolerance is zero; otherwise measure and document the accepted RPO.
Should failback happen in the same drill?
- Yes, to test the full cycle
- No, restore the topology in a later window
Test failback in the same drill once the team is ready; an untested failback is half a recovery plan.
Expected outcome
The team knows its real regional recovery time and data loss window, and can return to the original region without split-brain or lost writes.
Verification
Compare the post-failover checksum with the baseline, confirm a committed write survives failback, and verify the global endpoint and health checks reflect the active region.
Cleanup
Restore DNS and load-balancer weights, re-establish replication, remove temporary access, and update the runbook with measured timings.
A region failover is a system exercise
Moving traffic is the easy part. The application must connect to the promoted database, caches must be reachable, secrets must exist in the secondary region, background workers must resume, and DNS TTLs must not keep clients on the failed region. The drill tests the whole path, not only the load-balancer switch.
Baseline: primary eu-west-1, replica lag 200 ms, 4k rps
Failure: remove eu-west-1 from global endpoint at 10:00:00
Promotion: database read-write at 10:02:10
Traffic: 95% on us-east-1 at 10:04:00
Write check: order-7781 committed and read back
Failback: eu-west-1 caught up, traffic restored at 10:35:00Measure RTO and RPO
Recovery time objective is the elapsed time until the service is usable, not only until the database is promoted. Measure detection, decision, promotion, traffic shift, and application verification separately. Recovery point objective is the data loss window: compare the last committed write in the primary with the last replicated write in the secondary. A drill that reports only “failover succeeded” hides both numbers.
Prevent split brain
Do not allow both regions to accept writes for the same data. Promotion must be coordinated with fencing of the old primary, or the data layer must use a quorum. After the drill, inspect for divergent writes before restoring replication. A fast failover that creates two histories is worse than a slower, clean one.
Fail back deliberately
Failback is another migration. Rebuild the old region, restore replication, verify lag, shift a small amount of traffic, and then complete the move. Keep the rollback window open and monitor for errors that only appear with the original topology. Update the runbook with the measured times and every manual step.