阅读中文版本

Semantic Caching

Semantic caching returns a stored answer when a new prompt is close enough in meaning to a previous one, cutting cost and latency but risking subtly wrong reuse.

Definition

Semantic caching is an optimisation that embeds incoming requests and serves a cached response when a prior request is sufficiently similar, instead of always calling the model again.

It reduces spend and latency for applications with many paraphrased or repeated questions. Compared to simpler alternatives, this concept fits best when the trade-off it represents is the one the system actually cares about.

When to use it

A common use case is applying it at the boundary where the cost of getting it wrong is highest, and the scenario usually involves a concrete guarantee the system must uphold. Unlike an ad hoc approach, it makes that guarantee explicit, reviewable, and testable.

How it works

Because near-similar prompts can require different answers, the similarity threshold is a correctness knob: too loose and wrong answers are served, too tight and the cache barely hits.

Keys should include context that changes the answer, such as tenant, locale, or the freshness of the underlying data, to avoid cross-contamination.

Trade-offs and alternatives

Cache entries need expiry and invalidation, since the correct answer changes when source documents or policies change. The main drawback is the extra complexity it introduces, so the benefit should clearly outweigh that cost before a team adopts it.

Common misconceptions

A persistent misconception is that a high similarity score guarantees the cached answer still applies. Another is that exact string caching and semantic caching behave the same. For example, an engineer who assumes otherwise may design around a guarantee that does not actually exist, which only becomes obvious under load or failure.

Anti-patterns to avoid

Two recurring anti-patterns are to sharing one similarity threshold across unrelated topics and to caching answers that depend on time-sensitive or user-specific data. Both tend to surface as hard-to-debug production issues rather than obvious test failures. Related concepts include Document Chunking Strategies, AI Guardrails and Output Validation, Retrieval-Augmented Generation.