Batch vs Stream Processing
Batch processing runs bounded jobs over stored data; stream processing handles unbounded events continuously. Latency, cost, and correctness requirements decide the fit.
Definition
Batch processing reads a bounded input, computes a result, and writes output on a schedule. Stream processing consumes an unbounded sequence of events as they arrive, maintaining state and emitting results continuously. The split is about the shape of the input and the latency target, not the tool label.
Context and scope
Most pipelines combine both: streams for freshness and batch jobs for reprocessing, reconciliation, and complex historical analysis. Choosing one exclusively often creates either stale dashboards or fragile real-time logic.
Common misconceptions
- Stream processing is always more modern and should replace batch entirely.
- Batch jobs cannot handle event time or late-arriving data, so they are inherently less correct.
Anti-patterns
- Using a cron job every minute to simulate streaming when the source already emits events.
- Building a streaming pipeline with no replay path, so a bug requires manual data repair.
Related concepts
Related examples
Authoritative source
How it works
A batch job reads a finite dataset, often a partition of files or a database snapshot, processes it, and commits output. It can afford to sort, join, and retry the whole input, and it usually runs at a fixed cadence. A stream job reads events from a log or queue, keeps state for windows and joins, and emits results as records arrive. It must handle out-of-order events, restarts, and backpressure without losing progress.
The same computation can often be expressed in both models. A daily revenue report can scan all orders, while a streaming version maintains a running sum per minute and corrects it when late events appear.
When to use each
Use batch when latency of minutes or hours is acceptable, when the input is already stored, or when a full recomputation is the simplest correct answer. Use streaming when decisions depend on fresh events: fraud checks, live dashboards, alerting, or operational reactions. Many teams run a streaming path for freshness and a batch path as the source of truth for reconciliation.
Trade-offs
Batch is simpler to reason about and cheaper for large volumes, but results are stale by design. Streaming offers low latency but demands state management, watermarking, exactly-once or idempotent sinks, monitoring for lag, and a replay strategy. Streaming systems also tend to cost more at low volume because resources stay resident.
Example and misconception
A payments team streams authorizations to block suspicious cards within seconds, then runs a nightly batch to recompute settled revenue and detect discrepancies. The two paths share business rules but not execution.
The common misconception is that streaming means exactly-once by default. Delivery guarantees depend on source, state backend, and sink. A stream that processes an event twice can double-count unless the sink is idempotent or transactional. Batch has an analogous risk when a job reruns without replacing output atomically.