Data Corruption Incident Workflow
Freeze writes to affected data, establish the corruption window from logs and backups, and execute repair with row-level verification before unfreezing.
Objective
Contain a data corruption event, bound exactly which rows and which time window are wrong, and repair with verification before allowing writes again.
Environment
- A database with point-in-time recovery capability
- Row-level checksums or audit logs to detect corruption
- Authority to freeze writes on the affected table or service
Steps
1. Freeze writes and snapshot
Stop write traffic to the affected data (feature flag or read-only mode) and take an immediate snapshot of current state before any repair.
Expected state: Writes are stopped, and a forensic snapshot exists separate from live repair work.
2. Bound the corruption window
Use audit logs, replication lag history, and backup verification to determine the first and last corrupted write, producing a row-and-time window.
Expected state: A stated window exists: rows matching predicate X written between T1 and T2 are suspect.
3. Quantify blast radius
Count affected rows, identify downstream consumers of the corrupted values, and list derived data (caches, search indexes, exports) that ingested them.
Expected state: A blast-radius document names every store holding corrupted derivatives.
4. Repair with verification
Restore correct values from the pre-corruption backup or recompute them, writing a repair script that logs every changed row with before and after values.
Expected state: The repair log shows before/after per row, and a sample passes independent verification.
5. Unfreeze and purge derived copies
Resume writes gradually, replay queued valid writes, and purge or rebuild derived stores that hold corrupted values.
Expected state: Write traffic resumes, and no derived store still serves pre-repair corrupted values.
Expected outcome
A corruption event contained to a bounded window, repaired with per-row evidence, and purged from every derived store before full service resumes.
Verification
The forensic snapshot, the bounded window, the per-row repair log, and the derived-store purge list all exist in the incident record; sampled rows verify correct.
Cleanup
Retain the forensic snapshot and repair logs for the audit period; add the detection gap to the postmortem actions.
Corruption spreads while you discuss it
Every minute of continued writes after corruption extends the repair set and pushes bad values into derived stores. Freezing writes, snapshotting, and bounding the window converts an unbounded emergency into a finite repair job with evidence.
INC-2950 orders.total_cents corrupted by v1.8.2
14:12Z FREEZE writes on orders (flag orders-ro=on)
14:14Z forensic snapshot: orders_corrupt_20261022.pgdump
window (from audit log):
first bad row: 13:58:41Z (deploy v1.8.2 at 13:57)
last bad row: 14:11:59Z
predicate: total_cents = round(subtotal * 1.0)
count: 3,842 rows
blast radius: search index, nightly export, cache
repair: recompute from line_items, log per row
14:52Z 3,842 rows repaired, 50 sampled OK
unfreeze: 15:03Z gradual ramp
purge: search reindex 15:10Z, export re-run 15:40ZThe snapshot is the evidence, the repair is the treatment
Repairing in place destroys the evidence of what was wrong, which the postmortem and any audit will need. Snapshot first, repair second: the forensic copy preserves the failure for analysis while the live system moves toward correct.
Derived stores are the hidden second incident
The database can be perfect while search results, exports, and caches still serve corrupted values for weeks. Listing every derived consumer of the affected fields and purging each one is the difference between a closed incident and a slow-drip trust problem.