Late-Arriving Data Handling Workflow
Design pipelines that accept late events through event-time windows, watermark policies, and explicit restatement of already-published aggregates.
Objective
Keep late-arriving records from being silently dropped or double-counted, with published numbers restated deliberately when late data lands.
Environment
- A streaming or batch pipeline with event timestamps
- A defined lateness tolerance per data domain
- Downstream consumers who must be informed of restatements
Steps
1. Classify lateness tolerance per domain
Decide how late each data domain can be and still count: hours for ops metrics, days for financial reconciliation, never for sessionization keys.
Expected state: Each domain has a stated lateness window that drives the pipeline design.
2. Window on event time with watermarks
Group processing by event timestamp, advancing a watermark that declares how long the pipeline waits before closing a window.
Expected state: Windows close by event time, not by processing time, so late events find their true window.
3. Route by lateness class
Events within tolerance update their window; events beyond tolerance route to a late-arrival stream for review instead of being dropped.
Expected state: The out-of-tolerance stream is visible and queryable, not a metric on a debug page.
4. Restate published aggregates deliberately
When late data changes already-published numbers, recompute the affected windows and publish a restatement with a version marker that downstream consumers can distinguish.
Expected state: Consumers see v2 of an affected day and can tell it apart from the original.
5. Monitor lateness itself
Track the lateness distribution and the rate of out-of-tolerance events as pipeline health signals, alerting when sources drift late.
Expected state: Lateness metrics reveal upstream degradation before it becomes a correctness dispute.
Expected outcome
Pipelines that incorporate late data correctly, restate published numbers with visible versions, and detect upstream lateness drift as a monitored signal.
Verification
A test event injected 30 minutes late lands in its correct event-time window; an event beyond tolerance appears in the late-arrival stream; the affected daily aggregate republishes as a distinguishable version.
Late data is normal; silent wrongness is not
Sources deliver late: mobile offline events, partner batch files, clock skew. Pretending otherwise forces a choice between dropping data and double-counting it. Event-time windows, lateness routing, and deliberate restatement make late data a handled case.
domain: financial reconciliation
tolerance: 72h (event time)
10-22 run: event at 10-19 22:41Z arrives (46h late)
-> within tolerance -> joins 10-19 window
-> 10-19 daily total changes 41,203 -> 41,417
-> RESTATEMENT: daily_v2 published
consumers see version marker (v2)
10-23 run: event at 10-15 arrives (7d late)
-> beyond tolerance -> late-arrival stream
-> weekly review decides: manual adjustment
lateness monitor: p95 lateness 3.2h (baseline 1.1h)
-> source degradation alert firedProcessing-time windows misplace late events
Grouping by when data arrives puts a 46-hour-late event into today’s numbers, which are about yesterday — quietly wrong in a way reconciliation eventually catches loudly. Event-time windows put each event into the period it describes, which is the only definition that matches how the business thinks.
Restatements must be visible versions
Silently recomputing a published number means two consumers who looked at different times disagree forever, with no artifact explaining why. Publishing restatements as versioned values (day, total, version) makes the change a first-class fact downstream can track.