Rotate a Live API Key Without Downtime

Issue a second key, deploy dual-key acceptance, shift traffic to the new key, then revoke the old one after a quiet observation window.

~30 minsecuritysecretsoperations

Task

Replace an active API key used by clients and servers through an overlapping two-key window with no failed requests.

When to use this

A quarterly policy requires rotating the signing key for a partner integration. Naively replacing it would break every request during propagation.

Prerequisites

A credential store that can hold two active values per key slot
Control of the verification side and a coordination channel with the client
Monitoring that can attribute errors by key

Steps

1. Issue the new key alongside the old one.

aws secretsmanager create-secret \
  --name prod/payment-signing/v2 \
  --secret-string "$(openssl rand -hex 32)"

Expected output: The v2 secret exists while v1 remains active; nothing consuming v1 is affected yet.

2. Make the verifier accept both keys.

const accepted = await Promise.all([
  secrets.get("prod/payment-signing/v1"),
  secrets.get("prod/payment-signing/v2"),
]);
const valid = signatures.filter((s) => accepted.includes(s.key));

Expected output: Requests signed with either key validate during the overlap window, so propagation is invisible.

3. Switch the signer to the new key and watch traffic shift.

const signingKey = await secrets.get("prod/payment-signing/v2");

Expected output: Error-by-key metrics show new-key traffic rising and old-key traffic decaying as the client propagates the change.

4. Wait out the overlap window based on observed usage.

curl -s "http://metrics:9090/api/v1/query?query=signing_key_usage" | jq '.data.result[].metric.key' | sort | uniq -c

Expected output: Old-key usage reaching zero is the empirical signal that revocation is safe, better than any calendar guess.

5. Revoke the old key only after zero usage.

aws secretsmanager delete-secret \
  --secret-id prod/payment-signing/v1 \
  --recovery-window-in-days 7

Expected output: The old secret is scheduled for deletion with a recovery window, closing the rotation with a bounded undo.

Troubleshooting

Problem

Old-key usage never reaches zero

Solution

Some client caches the key forever; find it via per-key metrics, contact the owner, and only then extend the window rather than revoking blindly.

Problem

Verifier cannot hold two keys

Solution

Refactor to a key list before rotating; a single-slot verifier forces the big-bang rotation this recipe exists to avoid.

Problem

Emergency rotation after a leak

Solution

Compress the schedule but keep the shape: dual-accept, shift, revoke; skipping straight to revocation trades a planned window for a self-inflicted outage.

Limitations

The overlap window length is a business decision balancing availability against exposure of the old key.
Per-key telemetry is a prerequisite; without it, zero-usage is a guess and revocation is a gamble.

Rotation is a state machine, not a swap

Replacing a live credential atomically fails whenever two sides propagate at different speeds: the verifier learns the new key before the signer, or vice versa, and every request in between fails. The correct model is a state machine with an overlap state: both keys valid, traffic migrating, old key observed to zero, then revoked. Downtime disappears because no step ever requires the two sides to agree on timing.

// Overlap state: accept both
const accepted = await Promise.all([
  secrets.get("prod/payment-signing/v1"),
  secrets.get("prod/payment-signing/v2"),
]);

// Revoke only when observed usage hits zero
// metrics: signing_key_usage{key="v1"} == 0 for 24h

Let metrics end the overlap

The most common mistake is ending the window by calendar: rotate on Monday, revoke on Wednesday. The honest signal is per-key usage reaching zero and staying there. That requires telemetry keyed by credential, which is worth building once and reusing for every future rotation and for leak detection.

Keep the shape even in emergencies

A leaked key tempts you to revoke instantly, but the dual-accept shape compresses to minutes rather than disappearing: accept both, shift the signer, watch usage drop, revoke. The alternative revokes a key that 20 percent of traffic still signs with, and the incident becomes an outage you caused yourself.