Nanosecond Latency Budget
A nanosecond latency budget applies to the innermost loop — packet parsing, cache lookup — where every instruction is a measurable fraction of the budget and the design is cycle-accounted.
Definition
A nanosecond latency budget is the performance target for code paths where the per-operation cost is measured in nanoseconds — packet forwarding, cache lookup, interrupt handling — and the design accounts for every instruction, memory access, and branch in cycles against the budget.
Context and scope
Applied in high-frequency trading, kernel fast paths, and any code where the budget is a few hundred cycles and a cache miss is a budget overrun.
Common misconceptions
- A nanosecond budget is just low-latency — low-latency is microsecond; nanosecond is a different regime where a cache miss is catastrophic and the compiler is a collaborator
- Measurement at nanosecond resolution is impossible — CPU performance counters and cycle-accurate simulators measure at cycle resolution, which is sub-nanosecond
Anti-patterns
- Optimizing a nanosecond path with a microsecond profiler, whose overhead and resolution swamp the signal
- Counting instructions without counting cache misses, missing the cost that dominates the budget
Related concepts
Authoritative source
What it is
A nanosecond latency budget is the performance target for code paths where the per-operation cost is measured in nanoseconds — packet forwarding, cache lookup, interrupt handling — and the design accounts for every instruction, memory access, and branch in cycles against the budget. Applied in high-frequency trading, kernel fast paths, and any code where the budget is a few hundred cycles and a cache miss is a budget overrun. Performance is a distribution, not a number: the tail determines user experience, and the mean hides the outliers that cause abandonment.
How it works
Measurement targets percentiles at the workload’s actual concurrency, not a single sequential run. The fast path is kept fast by deferring work that is not on it — lazy initialization, coalescing writes, avoiding allocation in hot loops. Profiling finds the bottleneck; micro-optimization without a profile optimizes the wrong thing. Constant factors matter, but algorithmic complexity dominates as scale grows.
When to optimize and pitfalls
Optimize when a measurement shows a concrete cost — latency budget burn, compute spend, user-visible jank. The pitfall is optimizing for the benchmark rather than the workload: a synthetic test that does not reproduce the production access pattern produces a local maximum that hurts globally. Another is trading memory for speed without accounting for the GC or eviction cost the extra allocation introduces.