IO Batching Strategy
IO batching strategy groups multiple small operations into one large transfer, so the per-operation fixed cost — syscall, network round-trip — is amortized across the batch.
Definition
IO batching strategy is the design of grouping multiple small I/O operations — writes, sends, reads — into a single larger transfer, so that the fixed per-operation cost (syscall overhead, network round-trip, seek time) is paid once per batch rather than once per element, improving throughput at the cost of latency and complexity.
Context and scope
Applied in network I/O, disk writes, database inserts, and any workload with many small operations where the fixed cost dominates the per-element cost.
Common misconceptions
- Batching always improves throughput — it improves throughput at the cost of latency; for latency-sensitive workloads, the batch wait may not be worth it
- Larger batches are always better — they amortize more; past a point, the batch memory cost and the tail latency of a full batch outweigh the throughput gain
Anti-patterns
- Batching so aggressively that the batch wait adds latency beyond the SLO, trading throughput for SLO violation
- Flushing the batch on a timer without a size trigger, so a low-rate workload flushes tiny batches and gains nothing
Related concepts
Authoritative source
https://networkingnerd.net/2017/10/30/batching-vs-pipelining/
What it is
IO batching strategy is the design of grouping multiple small I/O operations — writes, sends, reads — into a single larger transfer, so that the fixed per-operation cost (syscall overhead, network round-trip, seek time) is paid once per batch rather than once per element, improving throughput at the cost of latency and complexity. Applied in network I/O, disk writes, database inserts, and any workload with many small operations where the fixed cost dominates the per-element cost. Performance is a distribution, not a number: the tail determines user experience, and the mean hides the outliers that cause abandonment.
How it works
Measurement targets percentiles at the workload’s actual concurrency, not a single sequential run. The fast path is kept fast by deferring work that is not on it — lazy initialization, coalescing writes, avoiding allocation in hot loops. Profiling finds the bottleneck; micro-optimization without a profile optimizes the wrong thing. Constant factors matter, but algorithmic complexity dominates as scale grows.
When to optimize and pitfalls
Optimize when a measurement shows a concrete cost — latency budget burn, compute spend, user-visible jank. The pitfall is optimizing for the benchmark rather than the workload: a synthetic test that does not reproduce the production access pattern produces a local maximum that hurts globally. Another is trading memory for speed without accounting for the GC or eviction cost the extra allocation introduces.