Watermarks
A watermark is a stream processor's estimate that no events earlier than a timestamp will arrive, letting it close windows and emit results with bounded delay.
Definition
In event-time stream processing, a watermark is a moving threshold that signals how far processing has advanced. When a watermark passes a window's end, the processor treats the window as complete and emits its result, while events older than the watermark are considered late.
Context and scope
Watermarks are the mechanism that makes event-time windows feasible on unbounded, out-of-order streams. Without them, a processor cannot know when a count for 10:00 to 10:05 is final.
Common misconceptions
- A watermark is a guarantee that no earlier event can ever arrive.
- Watermarks are wall-clock timers, so a slow machine delays window results.
Anti-patterns
- Setting allowed lateness to zero and silently dropping events that arrive slightly out of order.
- Using processing time for business windows when the event carries a meaningful timestamp.
Related concepts
Related examples
Authoritative source
https://nightlies.apache.org/flink/flink-docs-stable/docs/concepts/time/
How it works
Events carry an event-time timestamp. The processor tracks the maximum timestamp seen and emits a watermark, often as max_seen - allowed_delay. Operators compare the watermark with window boundaries. When the watermark passes the end of a window plus its allowed lateness, the window fires and its state can be released.
Watermarks flow through the dataflow alongside records. A union of streams takes the minimum watermark across inputs, because the slowest input may still produce old data. An idle partition can hold the minimum back forever, so engines expose idleness detection that temporarily excludes it.
Types of watermark
A perfect watermark knows that no future event can be earlier, which is only possible for bounded or coordinated sources. A heuristic watermark assumes bounded out-of-orderness and may be wrong. The heuristic is practical but means late events can still appear after a window fires. Some engines support punctuated watermarks driven by explicit markers in the stream.
When to use them
Use event-time windows and watermarks whenever the question is about when something happened rather than when it was processed: hourly sales, session durations, SLA compliance. Use processing time when freshness matters more than event ordering, such as detecting a sudden spike in logs, and accept that results depend on arrival time.
Trade-offs and example
Larger allowed delay produces more complete results but higher latency and more retained state. Smaller delay lowers latency but increases late arrivals. Watermarks also interact with exactly-once sinks: a window result emitted before late data arrives may need a retraction or an update.
Example: a job counts clicks per five-minute window with a 30-second watermark delay. At 10:05:30, the 10:00 to 10:05 window fires. A click timestamped 10:04:50 that arrives at 10:05:40 is late; with five minutes of allowed lateness, the job re-fires the window with a corrected count instead of losing the event.