Backup and Restore Drill

Prove that backups are usable by restoring them into an isolated environment, verifying data integrity, measuring recovery time, and documenting gaps.

intermediate~90 minbackupdisaster-recoverydatabases

Objective

Validate that the backup and restore process meets the recovery point and recovery time objectives before an incident requires it.

Environment

Automated database backups with known retention
An isolated restore environment with no production access
Documented recovery point objective and recovery time objective
A verification checklist for application-level integrity

Steps

1. Select a representative backup

Choose a recent backup and an older one from within the retention window. Record its timestamp and storage location.

Expected state: The chosen backup is within the documented retention and can be retrieved.

2. Restore into isolation

Restore to a separate database or cluster with production credentials removed. Do not overwrite production.

Expected state: The restore completes without modifying the production environment.

3. Verify database integrity

Check table counts, constraints, indexes, and a sample of critical records against known expectations or checksums.

Expected state: Data is complete and consistent for the checks performed.

4. Verify application behavior

Point a staging application at the restored database and exercise sign-in, read, and write journeys in the isolated environment.

Expected state: The application works against the restored data without errors.

5. Measure and document

Record restore duration, data loss window, manual steps, and any failures. Compare against the RTO and RPO.

Expected state: Actual recovery time and data loss are known, not assumed.

Decision points

How often should the drill run?

Quarterly with a representative backup
Annually

Run at least quarterly and after any backup or infrastructure change; annual drills miss drift.

What if the restore exceeds the RTO?

Treat it as an incident-level gap and remediate
Accept it as documentation only

A missed RTO is a real reliability gap; assign an owner and a deadline.

Expected outcome

A verified restore that meets or exposes gaps against RTO/RPO, with a documented procedure and owners for remediation.

Verification

Confirm the restored database passes integrity checks, the staging application completes critical journeys, and measured restore time is within the RTO with data loss within the RPO.

Cleanup

Destroy the isolated restore environment, remove temporary credentials, and archive the drill report and evidence.

A backup is not a recovery

A successful backup job proves only that files were written. A restore drill proves that those files can reconstruct a working system within the required time.

# Example: restore into an isolated database
createdb restore_drill
pg_restore --dbname=restore_drill --clean --if-exists backup.dump
psql restore_drill -c "SELECT count(*) FROM accounts;"

Drill sequence

Pick a backup inside the retention window.
Restore to an isolated environment.
Verify counts, constraints, indexes, and critical records.
Run application smoke journeys against the restored data.
Measure restore time and data loss; compare with RTO/RPO.

Verification

The restore completes without touching production.
Integrity checks pass.
The staging application can read and write against the restored database.
Restore time is measured and compared with the RTO.
Every gap has an owner and a remediation plan.

Common failures

Backups that exclude large tables, credentials that cannot be retrieved during an incident, and undocumented manual steps. A drill that always uses the same recent backup can also miss corruption that affects older restore points. Vary the selected backup and record the exact procedure used.