Schedule Automatic Database Secret Rotation

Enable Secrets Manager rotation on a 30-day schedule, make consumers reload credentials, and prove the old password stops working.

~40 minsecretsawsrotation

Task

Rotate production database credentials on a fixed schedule without manual tickets, downtime, or stale connection pools.

When to use this

A database password has not changed in two years and is stored in three services. A contractor with access to one deployment manifest could keep using the credential indefinitely, and no one knows how to rotate it safely.

Prerequisites

AWS Secrets Manager with an RDS PostgreSQL instance
The rotation Lambda for the database engine deployed in the same account
IAM permissions to create and rotate secrets
An application that can refresh a connection pool at runtime

Steps

1. Enable rotation with a Lambda function and a 30-day schedule.

aws secretsmanager rotate-secret \
  --secret-id prod/appdb/credentials \
  --rotation-lambda-arn arn:aws:lambda:us-east-1:123456789012:function:SecretsManagerRotation \
  --rotation-rules AutomaticallyAfterDays=30

Expected output: RotationEnabled becomes true and the secret gains AWSPENDING and AWSCURRENT version stages.

2. Confirm the schedule and the current version stages.

aws secretsmanager describe-secret --secret-id prod/appdb/credentials --query '{enabled:RotationEnabled,days:RotationRules.AutomaticallyAfterDays,versions:VersionIdsToStages}'

Expected output: enabled is true, days is 30, and exactly one version is marked AWSPENDING during an active rotation.

3. Refresh credentials on a timer or after an authentication failure instead of reading them once at boot.

async function currentCredentials() {
  const cached = await cache.get('db-creds');
  if (cached) return cached;

  const out = await secretsManager.getSecretValue({
    SecretId: 'prod/appdb/credentials',
  }).promise();
  const creds = JSON.parse(out.SecretString);
  await cache.set('db-creds', creds, { ttl: 300 });
  return creds;
}

pool.on('error', async (err) => {
  if (err.code === '28P01') {
    await cache.del('db-creds');
    await pool.end();
    pool = createPool(await currentCredentials());
  }
});

Expected output: A rotated credential is picked up within five minutes, and an authentication failure forces an immediate reload.

4. Rotate on demand and verify the old password is rejected before the change window ends.

aws secretsmanager rotate-secret --secret-id prod/appdb/credentials && psql 'postgresql://app_rw:[email protected]:5432/appdb' -c 'select 1;'

Expected output: The rotate call succeeds, the old password returns password authentication failed, and the application reconnects with AWSCURRENT.

Troubleshooting

Problem

Rotation fails with a password policy error in setSecret

Solution

Align the rotation Lambda password generation with the database parameter group policy, including length, character classes, and excluded characters.

Problem

The application keeps using the old credential after rotation

Solution

Check for credentials baked into environment variables or a pool created once at startup; switch to runtime fetch plus pool rebuild on 28P01.

Limitations

The rotation Lambda must support your engine and network path; private subnets need VPC access from Lambda.
Existing connections remain valid until the pool recycles, so rotation does not terminate sessions immediately.

Rotation without downtime

Secrets Manager rotation is a four-step Lambda flow: create a new secret version, set the new password on the database, test it, then move the AWSCURRENT label. Applications must read the secret at runtime and rebuild pools when authentication fails. Reading the credential once at boot turns a successful rotation into an outage.

Enable and observe

Start with a 30-day schedule. The first rotation runs immediately, which is a good way to validate the Lambda before trusting the timer. describe-secret shows RotationEnabled, the interval, and version stages. A version stuck in AWSPENDING after an hour means the Lambda failed midway; inspect CloudWatch Logs for the rotation function before retrying.

aws secretsmanager rotate-secret \
  --secret-id prod/appdb/credentials \
  --rotation-lambda-arn arn:aws:lambda:us-east-1:123456789012:function:SecretsManagerRotation \
  --rotation-rules AutomaticallyAfterDays=30

Make consumers reload

Cache the secret for a short TTL, such as five minutes, and refetch on authentication errors. On 28P01 invalid_password, discard the pool and build a new one from AWSCURRENT. Log the rotation event without logging the secret. If several services share the credential, all of them need the reload path before you enable rotation.

Verify the old credential

After a forced rotation, attempt a connection with the previous password and confirm it is rejected. Check that the new password works and that the application recovered without a restart. Keep a break-glass procedure: if the Lambda is broken, you still need a documented way to reset the credential and update the secret manually.