Run a Controlled Chaos Experiment

State a falsifiable resilience hypothesis, define steady state and blast radius, inject one failure, and abort automatically if the guardrail trips.

advanced~60 minreliabilitychaos-engineeringoperations

Objective

Learn whether a specific failure is handled as designed, with a bounded experiment that cannot silently degrade customers.

Environment

A staging or canary environment representative of production
A steady-state metric with a known normal range
A way to inject the failure (latency, error, pod kill, network partition)
An automatic abort condition and an on-call engineer watching

Steps

1. Write the hypothesis

State what should happen and how it will be observed, for example "killing one API pod keeps p95 under 300 ms and error rate under 0.1%."

Expected state: The hypothesis is falsifiable with a metric and a time window.

2. Define steady state and blast radius

Record the baseline metrics and limit the experiment to one region, one service, or a small percentage of traffic.

Expected state: The experiment has a measurable normal and a bounded scope.

3. Inject one failure

Apply the smallest injection that tests the hypothesis, with a time limit and an abort switch controlled by the observer.

Expected state: The failure is active only for the planned duration and only in the target scope.

4. Observe and abort

Watch the steady-state metric and the abort condition. Stop immediately if customer impact exceeds the guardrail, not only if the hypothesis fails.

Expected state: The observer can stop the experiment within seconds.

5. Restore and compare

Remove the injection, confirm the system returns to steady state, and compare the observed behavior with the hypothesis.

Expected state: Recovery is verified and the gap between design and reality is recorded.

6. Convert findings into work

File fixes for every unexpected behavior and add a regression test or runbook entry.

Expected state: The experiment produces owned changes, not only a report.

Decision points

Should the experiment run in production?

Staging first, then a small production canary
Only staging

Prove the abort mechanism in staging first, then run a bounded canary when production-specific behavior matters.

What stops the experiment?

The hypothesis fails
A customer-impact guardrail trips

Stop on either. A failed hypothesis is a learning; a guardrail breach is an incident.

Expected outcome

The team learns whether the failure is handled as designed, with no unbounded customer impact and with concrete follow-up work.

Verification

Compare the metric during and after the injection with the baseline; confirm the abort condition was tested separately and that the system recovered without manual data repair.

Cleanup

Remove injected faults, delete temporary network rules or chaos objects, restore capacity, and archive the experiment record.

Start with a falsifiable hypothesis

A chaos experiment is not random breakage. It tests a specific belief: “If one API pod dies, the service stays within its SLO.” The hypothesis names the failure, the expected behavior, the metric, and the observation window. If the result cannot be compared with the baseline, the experiment is a demo, not evidence.

Hypothesis: killing one of three API pods keeps p95 < 300 ms
Steady state: p95 180 ms, 5xx 0.02%, 3 pods ready
Blast radius: one pod in staging namespace
Abort: p95 > 500 ms or 5xx > 1% for 60 seconds

Bound the blast radius and the duration

Run in staging first. When production behavior matters, use a canary or a single zone and a short time limit. Define an abort condition in advance and give one observer the authority to stop. The abort should trigger on customer impact, not only on a failed hypothesis. A chaos experiment that causes an incident is itself a failure of the experiment design.

Restore and learn

Remove the injection and verify that the system returns to steady state. Compare the observed recovery time with the design. File work for every unexpected behavior: a missing timeout, a retry storm, a dashboard gap, or a runbook step that did not exist. The output is not the experiment report; it is the fixes and tests that make the next failure boring.

Do not use chaos as a substitute for design

Injecting failures cannot compensate for a missing timeout or a single-instance database. Fix the known fundamentals first. Chaos is most valuable when the system is already designed for failure and the team wants evidence that the design works under realistic conditions.