Protect a Redis Cache Against Stampede and Penetration

Guard a hot key with a mutex lock and stale-while-revalidate, and neutralize cache penetration for missing keys with short negative TTLs.

~35 mindatabaseredisresilience

Task

Prevent a hot cache key expiry from hammering the database, and stop queries for nonexistent keys from bypassing the cache entirely.

When to use this

A product page key expires every five minutes. At that instant, four hundred concurrent requests all miss, all query the database, and the database CPU saturates in seconds.

Prerequisites

Redis 6+ with SET NX support
A cache-aside read path in application code
A load test that can simulate concurrent misses on one key

Steps

1. Reproduce the stampede before fixing it.

hey -n 400 -c 100 -m GET "https://api.example.com/products/42" 2>&1 | grep -E "Status code|Average"

Expected output: Database query count spikes to hundreds during the window when the key expires, proving the herd is real.

2. Add a mutex so one request rebuilds the cache.

const LOCK = "lock:products:42";
if (await redis.set(LOCK, "1", { NX: true, EX: 5 })) {
  const data = await db.loadProduct(42);
  await redis.set(key, serialize(data), { EX: 300 });
  await redis.del(LOCK);
} else {
  await sleep(50); // brief wait, then re-read
}

Expected output: Only the lock holder queries the database; the other ninety-nine requests wait fifty milliseconds and read the fresh value.

3. Serve stale data while one worker rebuilds.

const raw = await redis.get(key);
if (raw) {
  const val = deserialize(raw);
  if (val.expiresAt < Date.now()) rebuildInBackground(key); // stale-while-revalidate
  return val.data;
}

Expected output: Readers never wait at all: slightly stale data is served instantly and one background job refreshes the key before it hard-expires.

4. Cache negative results to stop penetration.

const MISS = await db.loadProduct(4042);
if (!MISS) {
  await redis.set(key, serialize({ empty: true }), { EX: 30 }); // short negative TTL
  return null;
}

Expected output: Repeated lookups for a nonexistent id hit the cached empty marker for thirty seconds instead of the database every time.

5. Re-run the load test and count database queries.

hey -n 400 -c 100 -m GET "https://api.example.com/products/42" 2>&1 | grep "Status code"

Expected output: During expiry, exactly one database query runs per rebuild window and all other requests are served from cache.

Troubleshooting

Problem

Lock holder crashes before deleting

Solution

Always set EX on the mutex so the lock self-expires; a five-second lease bounds worst-case herd size even on crashes.

Problem

Negative TTL too long

Solution

A missing product that appears later stays invisible; keep negative TTLs short, tens of seconds, and purge them on create.

Problem

Many different keys expire together

Solution

Add jitter to TTLs so expirations spread out; synchronized expiry windows reproduce the stampede across the whole keyspace.

Limitations

Stale-while-revalidate serves outdated data briefly; only acceptable where seconds of staleness are tolerable.
The mutex adds a Redis round trip on the hot path; measure that it costs less than the herd it prevents.

One expiry, four hundred queries

A hot key with a TTL creates a synchronized thundering herd: at expiry, every concurrent request misses together and each independently rebuilds the same value. The mutex pattern lets exactly one request do the work while others wait briefly, and stale-while-revalidate removes even the waiting by serving the old value during rebuild. Negative caching closes the second hole, where lookups for keys that will never exist bypass the cache forever.

// Single-flight rebuild
if (await redis.set(LOCK, "1", { NX: true, EX: 5 })) {
  const data = await db.loadProduct(42);
  await redis.set(key, serialize(data), { EX: 300 });
  await redis.del(LOCK);
} else {
  await sleep(50);
}

// Negative cache for missing keys
if (!product) await redis.set(key, serialize({ empty: true }), { EX: 30 });

Lease the lock, never trust it

The mutex must carry an expiry of its own. A holder that crashes between acquiring and deleting would otherwise serialize all rebuilds behind a ghost lock forever. A five-second lease means the worst case is a short second herd, not a permanent outage, and the cost is one extra Redis command on the miss path only.

Jitter is the cheapest herd dispersal

If thousands of keys are written in one bulk load with identical TTLs, they expire in one synchronized wave and stampede together regardless of the mutex per key. Adding a random few percent to every TTL spreads expirations across a window, which costs nothing and removes the coordination the herd depends on.