Kappa Architecture
Kappa architecture replaces the batch-plus-speed lambda layers with a single stream-processing pipeline: one codebase, one storage, reprocessed by design instead of dual-run complexity.
Definition
Kappa treats all data as a stream: events land in an append-only log, stream processors compute views continuously, and reprocessing means replaying the log with a new processor version. It eliminates the lambda architecture dual codebase — batch and speed layers — and their reconciliation bugs, at the requirement that all logic be expressible as stream computation over a replayable log.
Context and scope
Adopted for analytics and feature pipelines where the log retains full history (Kafka with compaction or long retention). Jay Kreps coined it questioning the lambda.
Common misconceptions
- Kappa removes batch entirely — batch becomes a special case of streaming: replay the full log through the same processor
- Any workload fits kappa — workloads needing arbitrary SQL over full history and massive reprocessing windows still fit batch engines better
- The log must retain everything forever — retention policy matches the longest reprocessing window you accept; compaction keyed by entity bounds storage
Anti-patterns
- Rewriting processing logic without replaying history and having views that never recompute old events
- Running kappa on a log with short retention and losing the ability to backfill new views
- Keeping the lambda batch layer around just in case and paying dual-maintenance forever
Related concepts
Authoritative source
https://www.oreilly.com/radar/questioning-the-lambda-architecture/
What it is
The lambda architecture ran two systems for one dataset: a fast approximate layer and a slow accurate layer, reconciled at query time. Kappa observes that if the log is replayable, one stream processor is both — accuracy via replay, freshness via continuous processing.
How it works
Events append to a durable ordered log. Processors consume continuously, writing serving views. Reprocessing runs a new processor version over the log from offset zero in parallel with the current one, then cuts the serving layer over. The log is the system of record; views are disposable derivations.
When to choose
Pipelines dominated by event-shaped data, teams wanting one mental model and one codebase, and view logic that changes often enough that dual-layer maintenance hurts. Workloads with heavy ad-hoc reprocessing of arbitrary windows may still want a batch engine alongside.
Pitfalls
Reprocessing is O(log length): a long history with expensive transforms needs a re-derivation plan — checkpointing, parallelized partitions, hours-long windows. Some queries genuinely want ad-hoc scans rather than maintained views; a lakehouse query engine complements kappa rather than competing with it. Ordering and late events must be handled in the processor, since batch reconciliation no longer patches them.