Prefetch Strategies
Prefetch strategies issue memory loads before the data is needed, so the cache line arrives by the time the computation reads it, hiding the miss latency behind useful work.
Definition
Prefetch strategies are the techniques — software prefetch instructions, hardware prefetcher training via access pattern, and data layout that makes the pattern predictable — that load data into cache before the access, so that the computation does not stall on the miss and the latency is hidden behind independent work.
Context and scope
Applied in numerical loops, linked structure traversal, and any access pattern where the next address is knowable before the access.
Common misconceptions
- The hardware prefetcher handles everything — it handles sequential and strided patterns; for pointer-chasing and data-dependent addresses, software prefetch is needed
- Prefetch always helps — it helps when the prefetched line is used and the latency is hidden; a prefetch that evicts a useful line or is never consumed hurts
Anti-patterns
- Prefetching every access in a random pattern, issuing loads that are never used and wasting memory bandwidth
- Prefetching so far ahead that the line is evicted before it is used, defeating the intent
Related concepts
Authoritative source
https://software.intel.com/articles/software-optimization-for-intel-architecture
What it is
Prefetch strategies are the techniques — software prefetch instructions, hardware prefetcher training via access pattern, and data layout that makes the pattern predictable — that load data into cache before the access, so that the computation does not stall on the miss and the latency is hidden behind independent work. Applied in numerical loops, linked structure traversal, and any access pattern where the next address is knowable before the access. Performance is a distribution, not a number: the tail determines user experience, and the mean hides the outliers that cause abandonment.
How it works
Measurement targets percentiles at the workload’s actual concurrency, not a single sequential run. The fast path is kept fast by deferring work that is not on it — lazy initialization, coalescing writes, avoiding allocation in hot loops. Profiling finds the bottleneck; micro-optimization without a profile optimizes the wrong thing. Constant factors matter, but algorithmic complexity dominates as scale grows.
When to optimize and pitfalls
Optimize when a measurement shows a concrete cost — latency budget burn, compute spend, user-visible jank. The pitfall is optimizing for the benchmark rather than the workload: a synthetic test that does not reproduce the production access pattern produces a local maximum that hurts globally. Another is trading memory for speed without accounting for the GC or eviction cost the extra allocation introduces.