阅读中文版本

Late-Data Reprocessing Workflow

Reprocess late-arriving events into already-closed windows and correct downstream aggregates without double counting.

Overview

Correctly incorporate late data into closed windows and update downstream aggregates.

This workflow is reusable across teams and environments; adapt the commands to your tooling while keeping the sequence and checkpoints intact.

Environment and prerequisites

• A streaming or batch pipeline
• Idempotent window outputs
• Aggregate correction logic

Procedure

1. Detect late arrivals

Identify events whose event time falls into an already-emitted window.

Expected state: Late events are flagged rather than dropped.

2. Reprocess the affected window

Recompute the window including the late events using idempotent writes.

Expected state: The window reflects all events for its time range.

3. Correct downstream aggregates

Emit a correction or replace the affected aggregate values.

Expected state: Downstream totals match the corrected window.

4. Verify no double counting

Confirm repeated corrections do not inflate totals.

Expected state: Aggregates remain accurate across reprocessing.

# recompute a window idempotently
REPLACE INTO window_agg
SELECT window_start, SUM(value) FROM events WHERE window_start = :w GROUP BY window_start;

Verification

Confirm affected windows include late events, downstream aggregates match, and repeated reprocessing is stable.

Decision points

Emit corrections or rewrite windows?

• Emit idempotent corrections
• Rewrite and re-emit everything

Recommendation: Emit idempotent corrections to keep downstream updates bounded.

How late is too late?

• Define a lateness horizon
• Accept unlimited lateness

Recommendation: Set a lateness horizon and handle beyond-horizon data separately.

Common failures

• Dropping late events understates totals.
• Non-idempotent corrections double count.
• No horizon means unbounded reprocessing.