Cache Failure Graceful Degradation Workflow
Respond to cache layer failure by switching to degraded modes: stale-while-revalidate, direct origin with load shedding, and queue-based writes.
Objective
Keep the service usable during cache-layer failure by executing pre-built degradation modes instead of improvising under load.
Environment
- A cache layer (Redis, CDN, or in-memory) with failure modes identified
- Degradation flags or config switchable without deploy
- Origin capacity headroom measured in advance
Steps
1. Recognize the failure class
Distinguish cache outage (connection refused, cluster down) from cache slowness (timeouts) from cache corruption (wrong values), since each has a different degradation path.
Expected state: The incident record names the failure class, not just cache is broken.
2. Serve stale where correctness allows
For data tolerating staleness, switch to serving last-known values with a stale marker rather than erroring to users.
Expected state: Users receive slightly old data with a visible freshness indicator instead of errors.
3. Shed load to protect the origin
Before letting traffic hit the origin directly, enable request coalescing and load shedding on expensive endpoints, using the pre-measured origin headroom numbers.
Expected state: Origin CPU and latency stay within safe bounds under direct traffic.
4. Queue or reject non-critical writes
Route non-critical write traffic (analytics, counters) to a durable queue; reject or defer heavy write paths that the degraded mode cannot serve.
Expected state: Write paths either land durably or fail fast with clear messages, never half-process.
5. Recover and drain carefully
On cache recovery, warm critical keys, replay queued writes with idempotency keys, and ramp traffic back in steps while watching origin load.
Expected state: Traffic returns to full service without a recovery-induced second incident.
Decision points
How stale is acceptable for which data?
- Minutes for catalog, seconds for inventory
- Always fresh or error
Classify data by staleness tolerance in advance; binary fresh-or-error turns a cache blip into an outage.
Recovery: warm first or ramp first?
- Warm critical keys then ramp
- Ramp and absorb misses
Warm first for known hot sets; pure ramping turns cache misses into an origin stampede during recovery.
Expected outcome
Cache-layer failures that degrade to measurable modes with stale serving, protected origins, and durable writes, avoiding both outage and recovery backfire.
Verification
A chaos drill kills the cache layer and the service stays within its degraded SLO; recovery replay processes queued writes exactly once.
The cache is down; the service does not have to be
When the cache layer fails, the naive default is every request hitting the origin, which converts one failure into two. Pre-built degradation modes — stale serving, load shedding, queued writes — keep users in a slower but working service.
degradation ladder (cache failure):
L0 normal: cache hit 96%, origin 4%
L1 slow cache: timeouts -> stale-while-revalidate
L2 cache down: serve stale (catalog: 10 min)
+ coalesce misses (5s window)
+ shed heavy endpoints (p99 guard)
L3 origin risk: queue non-critical writes
+ defer batch jobs
chaos drill result (L2, 15 min):
availability 99.4% (SLO floor 99.0%)
origin p99 640ms (guard 800ms)
user impact: slower, stale-tagged, no errorsStaleness budgets are set before the incident
Deciding that inventory may be three seconds stale but payments must be exact is a product decision that cannot be made intelligently at 3 a.m. under load. The classification belongs in a table the degradation flags can reference.
Recovery needs the same care as failure
The classic second incident is cache recovery: cold cache, full traffic, origin stampede. Warming hot keys and ramping in steps, with the queued writes draining through idempotent replay, is what makes the end of the incident boring.