Resilience Verification Workflow

Verify resilience mechanisms (timeouts, retries, circuit breakers, fallbacks) by injecting the faults they exist for, on a schedule, in production-like conditions.

advanced~40 mintest-strategyresiliencechaos-engineering

Objective

Prove that resilience mechanisms actually function under the faults they were built for, rather than trusting configuration that has never fired.

Environment

A fault-injection capability (chaos tooling, proxy, or library)
Pre-production environment with production-like topology
Alerting on the mechanisms under test

Steps

1. Inventory the resilience mechanisms

List every timeout, retry policy, circuit breaker, bulkhead, and fallback with where it lives and what fault it addresses.

Expected state: An inventory maps each mechanism to its intended fault.

2. Design one fault per mechanism

For each mechanism, design the injection that should trigger it: latency above timeout, error rate above breaker threshold, dependency blackout for fallback.

Expected state: Each mechanism has a matching fault experiment with expected observables.

3. Inject in production-like conditions

Run the injections in an environment with real dependencies and real load shape, watching the mechanism fire through metrics rather than logs alone.

Expected state: Breakers visibly open, retries visibly back off, fallbacks visibly serve.

4. Verify user-visible behavior

During each injection, verify what users experienced: degraded-but-working paths, correct error surfaces, no silent data loss.

Expected state: Each experiment records the user-facing outcome alongside the mechanism trace.

5. Schedule and alert on decay

Repeat the experiments on a schedule, alerting when a mechanism stops responding to its fault — resilience rots as code changes.

Expected state: A missed fire in a scheduled run creates a ticket with the owning team.

Expected outcome

Resilience mechanisms verified to fire under their target faults, user experience during faults documented, and decay caught by scheduled re-verification.

Verification

The latest scheduled run shows every mechanism firing; disabling a breaker in a test is detected by the next run as a miss.

A breaker that has never opened is a hypothesis

Every timeout and circuit breaker in the codebase is a claim about behavior under a fault that may never have occurred. Injecting that fault — on a schedule, in production-like conditions — converts the claim into a verified fact or a discovered lie.

mechanism inventory -> experiment:
  payments timeout 800ms
    fault: inject 2s latency on payments calls
    expect: p99 <= 900ms, timeout errors < 1%
    observed: p99 3.1s   FAIL (timeout not wired!)

  search breaker (50% err threshold)
    fault: inject 60% errors for 60s
    expect: breaker opens ~10s, fallback served
    observed: opened at 11s, fallbacks 100%   PASS

  catalog fallback (static cache)
    fault: full blackout 5 min
    expect: static catalog served, no 5xx
    observed: static served, 0 errors        PASS

schedule: monthly; decay alert on missed fire

The never-fired mechanism is the risk

The payments timeout that failed verification had been configured for two years and never tested: under real latency it simply was not wired to the call path. Unused mechanisms are where confidence and reality diverge most, and only injection reveals it.

Resilience rots silently

A refactor moves a call off the instrumented client; a dependency rename breaks the breaker scope; the fallback cache expires empty. Scheduled re-injection with alerts on missed fires catches this decay the way scheduled backups-with-restore-tests catch untestable backups.