Configure Envoy Retries with a Budget

Enable Envoy retries with a retry budget and per-try timeout so transient failures recover without amplifying load on a struggling upstream.

~35 minenvoyreliabilityservice-mesh

Task

Add safe retries at the Envoy proxy with a budget that caps retry traffic.

When to use this

Retries are either absent so a blip fails the request, or unbounded so a struggling upstream gets hit harder by its own retries.

Prerequisites

Envoy proxy in the request path
Route configuration access

Steps

1. Enable retry_on for safe idempotent conditions and set num_retries low.

Expected output: Transient failures recover without duplication.

2. Set a retry budget so retries stay a small fraction of base traffic.

Expected output: Retries cannot sustain an overload.

3. Set per_try_timeout shorter than the overall timeout.

Expected output: Each attempt has its own deadline.

4. Restrict retries to idempotent methods to avoid duplicate writes.

curl -s http://localhost:9901/config_dump | grep -A5 retry_policy

Expected output: Non-idempotent calls are not retried blindly.

Troubleshooting

Problem

Retry storm on outage

Solution

Use the retry budget plus an outlier detector or circuit breaker; retries alone worsen outages.

Problem

Duplicate writes

Solution

Only retry idempotent methods; use idempotency keys for writes that need retries.

Limitations

Envoy retries only cover the proxy hop and its view of failures, not application-level errors.
A retry budget is a load control, not a correctness guarantee; consumers still need idempotency.

Context

Add safe retries at the Envoy proxy with a budget that caps retry traffic. Retries are either absent so a blip fails the request, or unbounded so a struggling upstream gets hit harder by its own retries.

Steps

Enable retry_on for safe idempotent conditions and set num_retries low. Expected: Transient failures recover without duplication.
Set a retry budget so retries stay a small fraction of base traffic. Expected: Retries cannot sustain an overload.
Set per_try_timeout shorter than the overall timeout. Expected: Each attempt has its own deadline.
Restrict retries to idempotent methods to avoid duplicate writes. Expected: Non-idempotent calls are not retried blindly.

route:
  retry_policy:
    retry_on: "5xx,connect-failure,reset"
    num_retries: 2
    per_try_timeout: 0.5s
    retry_budget:
      budget_percent: { value: 20 }

Verify

Transient failures recover without duplication.
Retries cannot sustain an overload.
Each attempt has its own deadline.
Non-idempotent calls are not retried blindly.

Limits

Envoy retries only cover the proxy hop and its view of failures, not application-level errors.
A retry budget is a load control, not a correctness guarantee; consumers still need idempotency.