Pipeline Failure Replay Workflow
Recover from pipeline failures by replaying affected windows with idempotent tasks, preserving inputs and skipping already-completed stages.
Objective
Turn pipeline recovery into a bounded replay of the failed window instead of a full re-run or manual patching of outputs.
Environment
- A pipeline whose tasks are idempotent or version-stamped
- Inputs retained for the replay window
- A record of which stages completed before the failure
Steps
1. Record completion per stage
Track which stages completed for which windows, so recovery knows exactly where the failure interrupted progress.
Expected state: A completion ledger exists mapping stage, window, and status.
2. Freeze the failed window
On failure, mark the affected window as needs-replay, preventing downstream consumers from reading the partial state.
Expected state: The window is flagged and consumers see a pending marker, not half-written data.
3. Replay from the failed stage
Re-run the window starting at the interrupted stage, reusing completed upstream outputs and re-executing from the failure point.
Expected state: The replay skips completed stages and the recovery time reflects one stage, not the whole pipeline.
4. Verify idempotency at merge points
For tasks that write rather than replace, ensure replay uses upserts keyed by window and entity so re-execution neither duplicates nor drops rows.
Expected state: A deliberately double-replayed window produces byte-identical output.
5. Close with reconciliation
After replay, reconcile the recovered window (counts, totals, key invariants) before clearing the needs-replay flag.
Expected state: The window closes only with the same checks a normal run passes.
Expected outcome
Pipeline failures recovered by bounded window replays that skip completed work, merge idempotently, and close with reconciliation evidence.
Verification
A drill that kills the pipeline mid-stage recovers by replaying only that stage; replaying the same window twice yields identical warehouse state.
Recovery should cost one stage, not one night
The crude recovery options are both expensive: re-run the whole pipeline, or hand-patch the outputs. Completion tracking plus window-scoped idempotent replay makes recovery cost exactly the interrupted stage.
ledger (stage x window):
extract 10-22 done 00:33Z
clean 10-22 done 00:36Z
enrich 10-22 FAILED 00:39Z <- interrupted
load 10-22 pending
recovery 09:12Z:
replay 10-22 from enrich (reuse extract+clean outputs)
enrich 00:39 -> 09:14 (2 min, 1 stage)
load 09:14 -> 09:15
reconcile: rows 2.0M, total matches source +/- 0
flag cleared 09:16Z
cost: 4 minutes (vs 41 min full re-run)Idempotency is the replay license
Re-running a stage that appends rather than upserts duplicates rows; a stage that reads mutable upstream state may see different inputs. Replay is only safe where every task is a function of (window, inputs), merging by key — which is also exactly what makes the double-replay test pass.
Freeze before replay, verify after
Partial state must be invisible to consumers during recovery, or reports generated mid-repair become wrong artifacts people screenshotted. The pending flag plus post-replay reconciliation bounds what any consumer could have seen.