Late Data
Late data is an event that arrives after its event-time window was expected to close. Handling it requires allowed lateness, retractions, or reprocessing.
Definition
Late data is a record whose event-time timestamp falls before the progress point of a stream, typically after a window has already emitted. It can result from network delay, mobile offline queues, producer backlog, or clock issues, and it forces a choice between dropping, updating, or reprocessing.
Context and scope
Any event-time pipeline encounters late data. The business impact ranges from a slightly delayed dashboard to an incorrect invoice, so the handling policy belongs in the design rather than in an operator's judgment.
Common misconceptions
- Late data is always a bug in the producer that should be rejected.
- Increasing the watermark delay eliminates late data entirely.
Anti-patterns
- Silently dropping late events without a metric or dead-letter path.
- Emitting only append-only aggregates when downstream consumers need corrected totals.
Related concepts
Related examples
Authoritative source
https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/operators/windows/
How it works
A stream processor advances an event-time watermark based on the maximum timestamp observed. When a window fires, its result is emitted. An event that arrives afterward and belongs to that window is late. The processor can drop it, accept it within an allowed-lateness period and re-fire the window, route it to a side output, or leave it for a batch correction job.
Late data is not the same as out-of-order data. Events can arrive out of order before the watermark closes the window and still be counted normally. The problem begins when the window has already produced output, because downstream systems have consumed a result that is now incomplete.
Choosing a policy
First measure: count late events, their lag distribution, and which sources produce them. If the lag is seconds and the business tolerates a small correction, allow lateness and emit updated results with a clear retraction or upsert key. If the lag can be hours, use a batch reconciliation path or a mutable serving table that accepts late corrections. If late events are unacceptable, enforce producer timestamps and reject events outside a defined horizon.
Whatever the policy, make it observable. A silent drop is indistinguishable from a source that never sent the event.
Trade-offs
Allowing lateness increases state size and result churn, because windows stay open and may emit multiple times. Dropping keeps state small but loses accuracy. Reprocessing a batch is accurate and simple to reason about but adds latency and duplicate-emission risk if the output is append-only.
Example and misconception
A mobile app queues check-ins while offline and uploads them when connectivity returns. A pipeline that drops anything older than one minute loses legitimate visits. A better design keeps a 24-hour allowed-lateness window, writes results as upserts keyed by window and store, and runs a nightly batch to settle the day.
The misconception is that late data is rare or malicious. In practice, ordinary mobile networks and producer retries make it a normal operating condition.