Shadow Traffic

Shadow traffic mirrors real production requests to a new service without returning its responses, validating behavior under real load before users depend on it.

delivery

Definition

Shadow traffic (traffic mirroring) copies production requests and sends them to a new implementation while the original response continues to serve the user. The shadow service must suppress side effects and avoid returning data to the client. The team compares responses, latency, and error rates under real traffic before switching any user.

Context and scope

Shadowing is used for migrations, rewrites, and performance validation where synthetic load cannot reproduce the real request mix, data shapes, or dependencies.

Common misconceptions

Mirroring is risk-free because users do not see the response — shadow writes, emails, and charges can still happen
Shadow traffic doubles capacity automatically — the shadow service needs its own capacity and dependency limits
A matching response proves equivalence — timing, side effects, and downstream state also matter

Anti-patterns

Sending real payment or email requests to the shadow service
Mirroring 100% of traffic before the shadow service has capacity
Ignoring personally identifiable data in shadow logs

Related concepts

Authoritative source

https://istio.io/latest/docs/tasks/traffic-management/mirroring/

Copy the request, not the side effects

A proxy can mirror a percentage of requests to a shadow service. The user still receives the original response. The shadow service processes the request against a safe data store, returns a response that is logged and discarded, and must not write to production tables, send messages, or call external providers.

route:
  - match: { prefix: /api/orders }
    route:
      - destination: { host: orders-v1 }
    mirror:
      host: orders-v2
    mirrorPercentage: 10

The example is an Istio-style route. The key is a bounded mirror percentage and a separately resourced destination.

Isolate writes and data

Shadowing a read endpoint is straightforward; shadowing a write is not. Use a sandbox database, a dry-run mode, or a separate account that cannot perform external effects. If the shadow service calls a payment provider, point it at a test provider or block the call. Redact or hash personal data in shadow logs, and apply the same access controls as production.

Compare under real load

Log the shadow response, status, and latency, then compare with the original. A shadow service that returns 200 but takes three times as long is not ready. Monitor the shadow service’s CPU, dependency usage, and error rate. Stop mirroring if it degrades the primary path or exhausts shared dependencies.

When to use it

Use shadow traffic when synthetic tests cannot reproduce the real request mix and a new implementation needs production evidence before exposure. The advantage is realistic validation without user impact; the drawback is duplicate load, side-effect risk, and data-privacy care. Compared to a canary, shadowing hides the new response entirely; compared to dark launch, it specifically mirrors live traffic.