Chaos Engineering
Chaos engineering runs controlled failure experiments in production-like systems to validate resilience assumptions before real outages expose them.
Definition
Chaos engineering is the disciplined practice of injecting realistic failures, such as latency, instance loss, or dependency errors, under controlled conditions to test whether a system maintains its promised behavior.
Context and scope
Applied to distributed systems whose redundancy and observability are mature enough to detect and abort an experiment safely.
Common misconceptions
- Chaos engineering is random breakage or a publicity stunt
- It requires a large company with dedicated tooling
- It replaces unit, integration, or load testing
Anti-patterns
- Injecting faults in production without an abort condition or monitoring
- Running experiments with no hypothesis or steady-state metric
- Testing in an environment that shares nothing with production
Related concepts
Related examples
Authoritative source
How it works
Practitioners define a steady-state metric, state a falsifiable hypothesis, limit the blast radius, and inject a realistic fault such as latency, packet loss, or instance termination. The experiment runs with an abort condition and monitoring, then produces evidence: did the system maintain its promise? Experiments graduate from staging to production as confidence grows.
When to use it
Use it before peak traffic events, after architecture changes, and whenever failover claims are untested. It requires redundancy, observability, and the ability to stop the experiment quickly. Compared with unit or load testing, chaos engineering tests the interaction of real components, but it cannot prove correctness and carries real operational risk.
Trade-offs
The practice costs tooling, training, and coordination, and a poorly scoped experiment can hurt customers. Results are probabilistic: one clean run does not prove resilience. Teams need approval paths, off-hours safeguards, and a culture that treats a discovered weakness as a success rather than a failure.
Example
During business hours, a team terminates ten percent of pods in one availability zone. The hypothesis is that p99 latency stays below 300 milliseconds and error rate below 0.1 percent. The abort condition fires when either is breached, and the experiment reveals a missing retry policy before the holiday peak.
Misconceptions
Chaos engineering is not random breakage or a stunt; it is a controlled experiment with a hypothesis. It does not require a huge company or a dedicated tool, because a manual game day works. And it does not replace testing; it tests assumptions that tests cannot reach.
Related concepts
Postmortems turn experiment findings into fixes; SLOs define the steady state to protect; incident command coordinates the response if an experiment escapes its bounds.