Disaster Recovery Drill Workflow
Declare a region failed, fail over to the standby using the runbook, measure the recovery time and data loss, and feed the gaps back into the runbook before the next drill.
Objective
Prove the disaster recovery plan works by failing over in a drill, measuring RTO and RPO, and fixing every gap the drill exposes.
Environment
- A primary region and a standby region with replicated data
- A runbook that a human can follow without the primary
- A way to route traffic to the standby without the primary
Steps
1. Declare the drill
Announce the drill window, confirm the standby is in sync, and arm the traffic cutover.
Expected state: The team knows the drill is on, the standby lag is near zero, and the cutover is ready.
2. Fail over
Follow the runbook to route traffic to the standby, promote the standby to primary, and confirm the app serves.
Expected state: The standby is primary, the app serves from it, and the dashboard shows traffic on the new region.
3. Measure RTO and RPO
Record the time from failure to serving and the data loss from the last replicated write, and compare to the targets.
Expected state: An RTO and RPO number per drill, with a pass or fail against the targets.
4. Feed gaps back
List every step that was unclear or slow, and file a follow-up to fix the runbook before the next drill.
Expected state: The runbook is updated, the follow-ups are tracked, and the next drill is scheduled.
Decision points
Surprise or announced?
- Announced for the first drill
- Surprise once the runbook is proven
Announced first; surprise drills only after a few announced ones pass, so you do not fail a real incident.
Fail back or stay?
- Fail back to the original primary
- Stay on the standby
Fail back for the drill; staying is a migration, not a drill.
Expected outcome
A successful failover with measured RTO and RPO, and a runbook improved by every gap the drill exposed.
Verification
Confirm the app served from the standby, the RTO and RPO are within target, and every runbook gap has a follow-up.
Cleanup
Restore the primary or promote the standby permanently, archive the drill artifacts, and schedule the next drill.
The runbook is the product
A DR plan you have never run is a hope. Run the runbook in a drill, measure the time and the data loss, and fix every step that was unclear. The runbook gets better every drill.
# cutover traffic to standby
dns update --region standby --ttl 60Measure RTO and RPO
RTO is the time from failure to serving. RPO is the data lost since the last replicated write. Record both per drill and compare to the targets. A drill that misses the target is a finding, not a failure.
Feed the gaps back
Every step that was unclear or slow becomes a follow-up. The runbook is updated before the next drill. Schedule the next drill before the follow-ups go stale.
Verification
The app served from the standby, the RTO and RPO are within target, and every runbook gap has a follow-up. Start with announced drills; move to surprise drills only after a few pass. Fail back after the drill; staying is a migration.