语义缓存

概念概述

语义缓存（Semantic Caching）指的是：语义缓存在新提示与既往提示含义足够接近时返回已存答案，降低成本与延迟，却有复用错误答案的风险。

从更严格的定义看，Semantic caching is an optimisation that embeds incoming requests and serves a cached response when a prior request is sufficiently similar, instead of always calling the model again. 该概念在工程实践中的价值，在于它能帮助团队用统一的语言描述一个具体的质量属性或设计约束，从而在评审与协作中减少歧义。

为什么需要它

It reduces spend and latency for applications with many paraphrased or repeated questions. 它与 Document Chunking Strategies、AI Guardrails and Output Validation、Retrieval-Augmented Generation 等概念经常一起出现，用于解决相近的工程问题。理解它的适用场景，有助于在多种备选方案之间做出有依据的取舍，而不是凭直觉选择技术方案。

常见误解

• a high similarity score guarantees the cached answer still applies
• exact string caching and semantic caching behave the same

需要避免的反模式

• sharing one similarity threshold across unrelated topics
• caching answers that depend on time-sensitive or user-specific data

实践要点

1. Because near-similar prompts can require different answers, the similarity threshold is a correctness knob: too loose and wrong answers are served, too tight and the cache barely hits.
2. Keys should include context that changes the answer, such as tenant, locale, or the freshness of the underlying data, to avoid cross-contamination.
3. Cache entries need expiry and invalidation, since the correct answer changes when source documents or policies change.

参考来源

https://en.wikipedia.org/wiki/Cache_(computing)

代码示例

interface CacheEntry<T> {
  value: T;
  expiresAt: number;
}

function Cached<T>(ttlMs: number) {
  return function (
    target: object,
    propertyKey: string,
    descriptor: TypedPropertyDescriptor<(...args: never[]) => Promise<T>>,
  ) {
    const original = descriptor.value!;
    const store = new Map<string, CacheEntry<T>>();

    descriptor.value = async function (this: unknown, ...args: never[]) {
      const key = propertyKey + ':' + JSON.stringify(args);
      const hit = store.get(key);
      const now = Date.now();

      if (hit && hit.expiresAt > now) {
        (this as { __cacheHits?: number }).__cacheHits ??= 0;
        return hit.value;
      }

      const value = await original.apply(this, args);
      store.set(key, { value, expiresAt: now + ttlMs });
      return value;
    };

    return descriptor;
  };
}

class ExchangeRates {
  private hits = 0;

  @Cached(30_000)
  async ratesFor(base: string): Promise<Record<string, number>> {
    const res = await fetch('https://api.example.com/rates?base=' + base);
    return res.json();
  }

  @Cached(300_000)
  async currencies(): Promise<string[]> {
    return ['USD', 'EUR', 'JPY'];
  }
}

const rates = new ExchangeRates();
const first = await rates.ratesFor('USD');
const second = await rates.ratesFor('USD');
console.log('cached identity:', first === second);