Scheduled Secret Rotation Workflow

Rotate credentials with overlapping validity, update consumers in a controlled order, verify access, and revoke the old version without downtime.

intermediate~60 minsecuritysecretsoperations

Objective

Replace a live credential on a schedule while every consumer keeps working and the old value stops granting access.

Environment

A secret manager that supports versioned secrets
Inventory of every service and job that reads the secret
Monitoring for authentication failures
Ability to redeploy or restart consumers

Steps

1. Inventory consumers

List every service, cron job, CI workflow, and third-party integration that uses the secret. Confirm each reads from the secret manager rather than a baked image or local file.

Expected state: Every consumer and its deployment path is known.

2. Create the new version

Generate a new credential and store it as a new secret version. Keep the old version valid during the overlap window.

Expected state: Both versions exist; the new one is not yet required.

3. Roll consumers in a controlled order

Redeploy or restart consumers in batches, starting with the least critical. Watch authentication errors after each batch.

Expected state: Each batch authenticates with the new version before the next batch starts.

4. Verify and revoke

Confirm the new credential works across all consumers, then disable or delete the old version. Monitor for late failures from forgotten consumers.

Expected state: The old credential is rejected and no consumer depends on it.

5. Record and schedule

Update the rotation log with time, owner, and evidence. Schedule the next rotation and review consumers that were discovered late.

Expected state: The rotation is auditable and the next window is known.

Decision points

Should both versions remain valid during rotation?

Yes, with a bounded overlap window
No, switch atomically

Use a bounded overlap for distributed consumers; atomic switching is only safe when every consumer can be updated simultaneously.

What triggers an emergency rotation?

Credential exposure or suspicious use
The scheduled interval

Exposure requires immediate revocation and rotation; do not wait for the schedule.

Expected outcome

A credential rotation completed with no authentication outage, every consumer on the new version, and the old value revoked.

Verification

Confirm the old credential fails from a test client, every consumer reports healthy authentication, and error dashboards show no sustained auth failures during the rollout.

Cleanup

Remove temporary dual-version configuration, update documentation, and delete the old secret version after the rollback window.

Overlap, then revoke

A scheduled rotation fails when one forgotten consumer still uses the old value. The safe pattern is to keep both versions valid for a bounded window, migrate consumers in batches, verify, then revoke.

create v2 -> deploy consumers in batches -> verify v2 -> disable v1 -> monitor

Checklist

Inventory every consumer.
Create the new version.
Roll out in batches, watching auth errors.
Confirm all consumers use the new version.
Disable the old version and monitor for late failures.
Record evidence and schedule the next rotation.

Verification

A test client using the old credential is rejected.
Every consumer reports successful authentication.
Error dashboards show no sustained auth failures.
The rotation log records the owner, time, and evidence.
The next rotation is scheduled.

Emergency rotation

If the credential was exposed, revoke it immediately rather than waiting for the overlap window. Accept a short outage if necessary, then rotate and update consumers under incident response. Scheduled rotations and incident rotations share the same checklist but differ in acceptable downtime.