Run a Controlled Latency Chaos Experiment

Inject 500ms of latency into one dependency with Toxiproxy during business hours, observe SLO burn, and verify timeouts and fallbacks.

~40 minchaos-engineeringlatencyresilience

Task

Validate that timeouts, retries, and dashboards behave correctly under a realistic dependency slowdown.

When to use this

Services have timeouts configured, but nobody has tested them. During a real slowdown, retries amplify load and the circuit breaker never opens because its threshold was never exercised against actual latency.

Prerequisites

Toxiproxy and toxiproxy-cli installed
A staging environment with the same client configuration as production
Dashboards for latency, error rate, and retry counts
A rollback plan and an experiment owner

Steps

1. Create a proxy in front of the dependency and point the staging client at the proxy port.

toxiproxy-cli create payments-db -l 127.0.0.1:26379 -u db.internal:5432

Expected output: Proxy payments-db listens on 127.0.0.1:26379 and forwards to db.internal:5432.

2. Add a 500 ms latency toxic with jitter and confirm the toxic is active.

toxiproxy-cli toxic add payments-db -t latency -a latency=500 -a jitter=50 && toxiproxy-cli inspect payments-db

Expected output: The inspect output lists latency_downstream with latency 500 and jitter 50.

3. Run a fixed load against checkout and watch p99, timeout rate, and retry count.

hey -z 5m -c 20 https://staging.example.com/checkout

Expected output: p99 rises by roughly the injected latency but stays within the timeout budget; retries remain bounded and fallbacks engage.

4. Remove the toxic, confirm recovery, and record the experiment result with any gaps found.

toxiproxy-cli toxic remove payments-db -n latency_downstream && curl -s -o /dev/null -w '%{time_total}\n' https://staging.example.com/checkout

Expected output: Latency returns to baseline within one minute, and the experiment report lists missing timeouts or retry storms.

Troubleshooting

Problem

All requests fail immediately during the experiment

Solution

The injected latency exceeds the client timeout; reduce to 300 ms to observe degradation rather than total failure, then test the timeout boundary separately.

Problem

The experiment affects production

Solution

Toxiproxy only proxies connections pointed at it; verify the staging client configuration and network policy before adding the toxic.

Limitations

Toxiproxy only manipulates TCP between the client and dependency; it cannot simulate CPU, disk, or code-level delays.
Staging traffic patterns differ from production, so results indicate behavior, not exact production impact.

Test the failure you designed for

Timeouts, retries, circuit breakers, and fallbacks are code paths that rarely run. A latency experiment forces them to execute under observation. The goal is not to break production but to learn whether the defenses behave as intended: retries bounded, timeouts firing, fallback serving a degraded response, and alerts firing at the right threshold.

Start with one dependency

Choose a dependency whose slowdown has a known blast radius, such as the payments database. Put Toxiproxy between the staging client and that dependency, then inject latency. Keep the experiment short and observe the SLO dashboards. Compare p99 latency with the configured timeout: if p99 approaches the timeout, the system is close to failure and the timeout may be too tight.

toxiproxy-cli create payments-db -l 127.0.0.1:26379 -u db.internal:5432
toxiproxy-cli toxic add payments-db -t latency -a latency=500 -a jitter=50

Watch the right signals

Track error rate, timeout count, retry count, and fallback usage, not just latency. A retry storm shows up as rising request volume against the dependency with little success. A circuit breaker that never opens suggests its failure threshold is too high. Record what you see; the experiment is only valuable if the observations turn into changes.

End cleanly

Remove the toxic, verify recovery within a minute, and close the experiment ticket with results. Schedule experiments during business hours with the team watching, not at night. If the experiment reveals a missing timeout or an unbounded retry, fix it and rerun the same experiment to confirm the behavior changed.