Pipes and Filters
Pipes and filters decompose processing into a chain of transforming components connected by pipes, enabling reconfiguration, parallelism, and independent evolution of each stage.
Definition
The pipes and filters pattern structures a system as a pipeline: filter components transform or filter their input, pipes transport data between them. Each filter is independent — it knows only its input and output contracts — enabling reuse, reordering, parallel fan-out, and separate deployment. Unix shell pipelines are the canonical instance; stream processors and build systems are industrial ones.
Context and scope
Applied in data ingestion pipelines, build toolchains, media processing, and stream processing topologies. Contrasts with the batch monolith doing all transforms in one program.
Common misconceptions
- Filters must coordinate with each other — coordination belongs in the data contracts; a filter knowing its neighbors breaks reconfiguration
- The pattern is only for streaming data — compilers, build systems, and CI are pipelines of exactly this shape
- More filters always improve clarity — every stage adds serialization, buffering, and failure surface; pipeline depth is a cost decision
Anti-patterns
- Passing giant mutable objects between filters and mutating them in place, corrupting stage independence
- Building pipelines with filters that fail silently and swallow data without error propagation
- Chaining filters synchronously for high-volume streams and serializing throughput to the slowest stage
Related concepts
Related examples
Authoritative source
https://learn.microsoft.com/en-us/azure/architecture/patterns/pipes-and-filters
What it is
Complex processing decomposes into stages, each doing one transformation well. The pipe is dumb transport; the filter is smart transformation. Composition happens by wiring, so the same filters build different pipelines.
How it works
Filters declare input and output schemas. Pipes move data — in-memory iterators, queues, or log topics. Stages scale independently: parallel instances of a slow filter multiply its throughput. Backpressure propagates naturally through bounded pipes. Error handling strategies: fail the pipeline, dead-letter the item, or route to a quarantine filter.
When to use it
Multi-step transformation with evolving steps: ETL, document processing, event enrichment, CI pipelines. Skip it when a single program with functions does the same job without transport overhead.
Pitfalls
Every pipe boundary costs serialization and copying — in-process pipelines should pass references where safe. Ordering requirements constrain fan-out: per-key parallelism preserves entity order. Schema evolution between filters needs contract versioning, or downstream filters break silently on upstream change.