Chaos Experiment Workflow
Define a steady state hypothesis, inject a failure in a blast-radius-bounded way, observe the system, and abort on SLO breach so the experiment never harms users.
Objective
Validate a resilience assumption by injecting a failure and confirming the system holds its steady state, without breaching the user-facing SLO.
Environment
- A staging or canary environment that mirrors production topology
- A chaos tool that can inject and halt failures
- A live SLO dashboard that updates within seconds
Steps
1. State the hypothesis
Write the steady state as a measurable condition, like error rate below 0.1 percent, and the failure you plan to inject.
Expected state: A hypothesis with a metric, a threshold, and a failure, all written down before the experiment starts.
2. Bound the blast radius
Limit the injection to a single instance or a canary slice, and confirm the auto-abort threshold is armed.
Expected state: The experiment can affect at most one slice, and the SLO breach triggers an automatic halt.
3. Inject and observe
Start the injection, watch the SLO dashboard, and record the system behavior at each second.
Expected state: The system either holds the steady state or the auto-abort fires within the threshold, never both ignored.
4. Halt and record
Stop the injection, confirm the system recovers, and record the result as evidence for or against the hypothesis.
Expected state: The injection is halted, the SLO recovers, and the experiment is logged with the hypothesis, the result, and the artifacts.
Decision points
Where do we run the experiment?
- Staging with production-like load
- A canary slice in production
Run in staging first; only move to a canary slice when staging has shown the result and the auto-abort is proven.
What if the hypothesis is wrong?
- File a follow-up and stop
- Fix it on the spot
File a follow-up; fixing under an armed injection risks a second failure while the first is still live.
Expected outcome
A recorded experiment that either confirms the resilience assumption or finds a gap, with the SLO never breached for users and the injection always halted.
Verification
Confirm the injection is halted, the SLO recovers to the steady state, and the experiment log contains the hypothesis, the result, and the dashboard snapshot.
Cleanup
Remove the injection hooks from the environment, archive the dashboard snapshot, and file a follow-up for any gap the experiment found.
Hypothesis before injection
A chaos experiment without a hypothesis is just a failure. Write the steady state as a metric and a threshold, and the failure you will inject, before you touch anything.
hypothesis: error_rate < 0.1% during single-instance failure
failure: terminate one api instance
abort: error_rate > 0.5% for 30sBound the blast radius
Inject into a single instance or a canary slice, never the whole fleet. Arm an auto-abort that halts the injection the moment the SLO breaches. The experiment must be safer than the failure it simulates.
Observe, then halt
Watch the SLO dashboard every second. If the steady state holds, record the win. If the auto-abort fires, record the gap and file a follow-up. Never leave an injection running while you debug.
Verification
The injection is halted, the SLO recovers, and the experiment log has the hypothesis, the result, and a dashboard snapshot. Run in staging first; move to a canary slice only when the auto-abort is proven.