阅读中文版本

Idempotent Pipeline Replay Workflow

Make a pipeline safe to replay by using deterministic keys and upserts so a rerun for the same window does not duplicate data.

Overview

Enable safe pipeline replays for any time window without duplicating data.

This workflow is reusable across teams and environments; adapt the commands to your tooling while keeping the sequence and checkpoints intact.

Environment and prerequisites

• A batch or streaming pipeline
• A warehouse with merge support
• Partition or window boundaries

Procedure

1. Define deterministic keys

Identify a stable key per record that is identical across runs.

Expected state: Every record has a replay-stable identity.

2. Write with upsert or merge

Use merge on the key so a rerun updates rather than duplicates.

Expected state: Reruns converge to the same table state.

3. Bound the replay window

Replay exact partitions and overwrite only the affected range.

Expected state: Replays touch only the intended window.

4. Verify row counts are stable

Rerun the same window and confirm counts and checksums do not change.

Expected state: Repeated runs produce identical results.

MERGE INTO events AS t
USING staging_events AS s
ON t.id = s.id
WHEN MATCHED THEN UPDATE SET ...
WHEN NOT MATCHED THEN INSERT ...;

Verification

Confirm rerunning the same window leaves row counts and checksums unchanged.

Decision points

Overwrite or merge?

• Merge on a stable key
• Append and dedupe later

Recommendation: Merge on a stable key so replays are correct by construction.

How to handle deleted source rows?

• Tombstone or soft delete
• Ignore deletions

Recommendation: Propagate deletions with tombstones so replay reflects reality.

Common failures

• Append-only writes duplicate on replay.
• Wide replay windows touch unrelated data.
• Ignoring deletions leaves stale rows.