Shadow Traffic Testing
Shadow traffic testing replays production requests against a candidate build without returning its responses, so the candidate is tested under real load with zero user impact.
Definition
Shadow traffic testing is the practice of mirroring production requests to a candidate deployment in addition to the live one, discarding the candidate responses, so that the candidate is exercised under real production load and its behavior, errors, and performance can be compared without any user seeing the candidate output.
Context and scope
Applied in pre-release validation, migration testing, and any change where the risk is behavior under real load that synthetic tests do not reproduce.
Common misconceptions
- Shadow traffic is just canary — canary serves the candidate to real users; shadow serves it without users, so the blast radius is zero
- Shadow catches all regressions — it catches behavior regressions under the mirrored load; it misses regressions on requests the mirror does not capture
Anti-patterns
- Mirroring traffic to a candidate that shares the production database, so a candidate bug writes corrupt data to production
- Mirroring without deduplicating, so the candidate does double the work and its performance metrics are not comparable
Related concepts
Authoritative source
https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/route/v3/route_components
What it is
Shadow traffic testing is the practice of mirroring production requests to a candidate deployment in addition to the live one, discarding the candidate responses, so that the candidate is exercised under real production load and its behavior, errors, and performance can be compared without any user seeing the candidate output. Applied in pre-release validation, migration testing, and any change where the risk is behavior under real load that synthetic tests do not reproduce. The unit of delivery is the deployable artifact plus its configuration, and the goal is to move changes to production safely, quickly, and reversibly.
How it works
Safety comes from small batches (less change per deploy means less blast radius), automated canaries (a bad change is caught at 1% before it reaches 100%), and reversible deployments (rollback is the default escape, not a rewrite). Speed comes from pipeline parallelism and eliminating the manual gates that batch up changes. Every step produces evidence — test results, scan output, canary metrics — that the next step consumes.
When to apply and pitfalls
Apply whenever the cost of an incident exceeds the cost of the pipeline that prevents it. The pitfall is optimizing deploy speed without investing in the observability and rollback that make speed safe — fast deploys of broken code just reach the outage sooner. Another is treating the pipeline as a checklist rather than a feedback loop that improves with each deployment.