← Concepts

阅读中文版本

Failure Path Testing

Failure path testing verifies degraded behavior deliberately: inject faults at dependencies, assert timeouts, retries, fallbacks, and partial states — production exercises these branches first.

testing

Definition

Failure path testing is the systematic verification of error handling: every external call gets a test where it fails — timeout, 5xx, malformed payload, connection reset — and the assertions target the recovery contract: retry policy, fallback content, circuit transitions, error surfaces to users, and cleanup of partial state. It complements chaos engineering at the unit and integration level.

Context and scope

Applied with fault-injection proxies (Toxiproxy), HTTP mocks returning failures, container kills, and framework-level test doubles for dependency failure.

Common misconceptions

  • Error handling is too defensive to test — untested error handling is where outages escalate: leaked locks, stuck retries, silent data loss
  • Catching the exception in a test proves the handler works — the assertion must verify the recovery behavior and state, not just absence of a throw
  • Failures are too diverse to cover — a small taxonomy — timeout, refusal, corruption, slowness — covers the design space of handlers

Anti-patterns

  • Testing only the happy path and shipping retry logic that has never executed
  • Asserting that a call throws without asserting resources were released and state rolled back
  • Injecting catastrophic failures only, missing the slow-degradation cases where most real bugs live

Related concepts

Related examples

Authoritative source

https://principlesofchaos.org/

What it is

Systems are designed to survive failures, then tested only for success. The survival code — timeouts, retries, fallbacks, cleanup — ships unexecuted until production runs it for the first time.

How it works

Enumerate the dependency failure taxonomy: connection refused, read timeout, write timeout, truncated body, semantic error, and slow-loris degradation. For each, inject at the boundary — proxy, mock, container — and assert the observable contract: which error surfaces, what state remains, whether retries respect budget and jitter, and whether the circuit opens.

When to invest

Before relying on any resilience mechanism in an incident: every timeout value and retry policy is a testable contract. Integration suites run the injected-failure matrix; chaos experiments verify the fleet-level composition of those behaviors.

Pitfalls

Fault injection at the wrong layer — mocking an exception the real dependency cannot produce — validates fiction. Timeout tests need controllable clocks or they run at real timeout duration and bloat the suite. Partial failure states — two-phase writes interrupted — need explicit assertions or corruption leaks through as a pass.