Coalesce Concurrent Requests for the Same Key

Merge simultaneous cache misses for one key into a single origin call, share the result, and clean up the in-flight entry so retries are possible.

~25 mincachingperformanceconcurrency

Task

Prevent a hot key from stampeding the origin when it expires or is first requested under load.

When to use this

A popular profile or report is requested hundreds of times per second. After a cache miss, every request starts the same expensive query.

Prerequisites

A process-local or shared cache
A stable cache key for the resource
A way to observe origin call count and latency

Steps

1. Implement a single-flight map that stores the in-flight promise per key and removes it in finally.

const inFlight = new Map<string, Promise<unknown>>();

export function singleFlight<T>(key: string, load: () => Promise<T>): Promise<T> {
  const existing = inFlight.get(key);
  if (existing) return existing as Promise<T>;

  const promise = load().finally(() => inFlight.delete(key));
  inFlight.set(key, promise);
  return promise;
}

Expected output: Concurrent calls for the same key share one promise and one origin request.

2. Combine coalescing with a cache read and a short TTL, and ensure the cache write happens once.

async function getProfile(id: string) {
  const cached = await cache.get(`profile:${id}`);
  if (cached) return JSON.parse(cached);

  const profile = await singleFlight(`profile:${id}`, () => db.loadProfile(id));
  await cache.set(`profile:${id}`, JSON.stringify(profile), { EX: 60 });
  return profile;
}

Expected output: A burst of misses produces one database query and one cache write.

3. Test with concurrent simulated requests and count origin invocations.

node --test coalescing.test.mjs

Expected output: Ten concurrent calls trigger exactly one loader invocation.

Troubleshooting

Problem

A failed load is cached forever

Solution

Delete the in-flight entry in finally so a later request can retry; do not store rejected promises.

Problem

Coalescing does not work across instances

Solution

The map is process-local. Use a distributed lease or a shared cache with stale-while-revalidate for cross-instance protection.

Limitations

Coalescing does not make the origin faster; it removes duplicate work for one key.
A slow loader holds all callers for that key; pair it with a timeout and a bounded wait.

One key, one origin call

When a hot key misses, concurrent requests all see the same miss. Coalescing stores the in-flight promise and returns it to every caller. The first request performs the load; the rest wait for the same result.

const existing = inFlight.get(key);
if (existing) return existing;
const promise = load().finally(() => inFlight.delete(key));
inFlight.set(key, promise);

Remove the entry on failure

Deleting in finally is essential. If a rejected promise stays in the map, every later request receives the same failure without retrying. Remove the entry and let the next request start a fresh load, subject to a timeout and a retry policy.

Bound the wait

A slow loader holds all callers for that key. Use the same timeout as the underlying dependency, and consider a short stale-while-revalidate cache so callers can receive an older value instead of waiting. Monitor coalesced wait time, origin call count, and error rate. A sudden rise in origin calls after deploying coalescing usually means the key includes a request-specific value and never matches.