中文
devexamples
← Workflows

阅读中文版本

Stream Processing Backfill Workflow

Backfill a derived stream from historical data, run it alongside live processing without corrupting state, and verify totals before promoting.

advanced~150 minstreamingdataoperations

Objective

Reprocess historical data into a derived stream or state store without disturbing the live pipeline.

Environment

  • A stream processor with checkpointing
  • A separate output namespace for the backfill

Steps

1. Validate the transformation on a small

Validate the transformation on a small historical sample.

Expected state: The logic is correct before a large backfill.

2. Run the backfill into a separate

Run the backfill into a separate output to avoid clobbering live results.

Expected state: Live processing is untouched.

3. Throttle the backfill so it does

Throttle the backfill so it does not starve live consumers.

Expected state: Shared resources stay healthy.

4. Compare backfill totals with expected values

Compare backfill totals with expected values and then promote.

Expected state: The output is proven before cutover.

Decision points

Backfill in the live state store or a separate one?

  • Separate
  • Live

Separate when the state store is not idempotent; overwriting live state risks corrupting running aggregates.

Reprocess all history or only a window?

  • Only what changed
  • All history

Reprocess only the affected window when the bug is bounded; full history is expensive and risky.

Expected outcome

The derived stream contains correct reprocessed data, live processing never saw corrupted state, and promotion is verified.

Verification

Check that backfill output totals match expected aggregates and that live processing metrics were unaffected during the backfill.

Cleanup

Remove the temporary output namespace after promotion and record the backfill job parameters.

Overview

Reprocess historical data into a derived stream or state store without disturbing the live pipeline. Check that backfill output totals match expected aggregates and that live processing metrics were unaffected during the backfill.

Steps

  1. Validate the transformation on a small historical sample. Expected state: The logic is correct before a large backfill.
  2. Run the backfill into a separate output to avoid clobbering live results. Expected state: Live processing is untouched.
  3. Throttle the backfill so it does not starve live consumers. Expected state: Shared resources stay healthy.
  4. Compare backfill totals with expected values and then promote. Expected state: The output is proven before cutover.
-- separate output for backfill
INSERT INTO metrics_rebuild
SELECT window, name, sum(value)
FROM events
WHERE ts >= $1 AND ts < $2
GROUP BY 1, 2;

Verification

  • The derived stream contains correct reprocessed data, live processing never saw corrupted state, and promotion is verified.

Decision points

  • Backfill in the live state store or a separate one? Options: Separate; Live. Recommended: Separate when the state store is not idempotent; overwriting live state risks corrupting running aggregates.
  • Reprocess all history or only a window? Options: Only what changed; All history. Recommended: Reprocess only the affected window when the bug is bounded; full history is expensive and risky.