Fault Injection
Fault injection deliberately introduces latency, errors, or resource failures to verify that a system detects, contains, and recovers from them.
Definition
Fault injection is the controlled introduction of faults into a system, its dependencies, or its infrastructure to observe how it behaves under failure.
Context and scope
Used in staging and, with safeguards, in production to validate timeouts, retries, failover, and degraded behavior.
Common misconceptions
- Fault injection requires breaking production to be realistic
- Fault injection and chaos engineering are interchangeable terms
- One clean experiment proves the system is resilient
Anti-patterns
- Injecting faults without a hypothesis, abort condition, or blast radius limit
- Testing only exceptions while ignoring latency and partial failure
- Injecting into an environment that shares none of production's dependencies
Related concepts
Related examples
Authoritative source
https://learn.microsoft.com/en-us/azure/chaos-studio/chaos-studio-overview
What it is
Fault injection introduces a specific failure on purpose: a dependency returns errors, a call takes three seconds instead of thirty milliseconds, a disk fills, a process dies, a network packet is dropped. The team then observes whether the system detects the fault, contains it, and recovers within its promises. Unlike waiting for an outage, injection makes the failure reproducible, attributable, and safe to study.
Fault types
Useful faults include latency, error responses, connection resets, resource exhaustion, clock skew, and partial data loss. Latency is often the most revealing because it triggers timeouts, retries, and queue growth that pure error injection does not. Combine faults: slow responses plus a retry storm can exhaust a connection pool that survives either fault alone. Match fault semantics to real dependency behavior rather than inventing convenient failures.
Levels
Injection can happen in test code through a wrapper that throws or delays, in the network through a proxy or service mesh, or at the infrastructure level by terminating instances or blocking zones. Lower levels are cheaper and more precise; higher levels test more of the real system. Start low, then graduate.
Safety
Define a hypothesis, a steady-state metric, a bounded blast radius, and an abort condition before starting. Never inject into a shared dependency without owner consent. In production, begin with a small percentage of traffic and off-peak hours. Record the experiment, the observations, and the follow-up actions.
Limits and relationship to chaos engineering
Fault injection is a technique; chaos engineering is the broader discipline of running experiments to validate resilience. A passing injection proves only that one fault, under one configuration, was handled. For example, killing a pod may succeed while a zone loss still fails. Use results to fix timeout budgets and failover paths, not to declare resilience complete.