Rotate TLS Certificates Without Downtime

Inventory certificates, issue replacements, deploy them with overlap, verify the served chain and expiry, and automate renewal before expiration becomes an outage.

intermediate~45 minsecuritytlsoperations

Objective

Replace TLS certificates before expiry or after a key compromise without breaking clients, APIs, or internal service-to-service calls.

Environment

An inventory of certificates, endpoints, and owners
Access to the certificate authority or ACME client
A deployment path that can update the certificate and reload the server
Monitoring for expiry, handshake failures, and trust-chain errors

Steps

1. Inventory and prioritize

List every certificate by endpoint, issuer, expiry, key type, and owner. Mark public, internal, and client certificates separately.

Expected state: Every certificate has an owner, an expiry date, and a rotation method.

2. Issue the replacement

Generate a new private key and certificate with the same subject alternative names. Use a key type and chain compatible with the oldest supported client.

Expected state: The new certificate validates against the intended chain and covers every required hostname.

3. Deploy with overlap

Install the new certificate on all servers, reload gracefully, and keep the old certificate available until connections drain.

Expected state: New connections use the new certificate while existing connections finish on the old one.

4. Verify from the outside

Check the served chain, expiry, hostname coverage, and protocol from multiple network locations. Test internal clients and pinned clients separately.

Expected state: External and internal checks show the new certificate with a valid chain.

5. Revoke and clean up

Revoke the old certificate if it was compromised, remove it from the servers, and update the inventory.

Expected state: The old certificate is no longer served and the inventory reflects the new expiry.

6. Automate the next rotation

Schedule renewal with a warning threshold and a dry run; alert if renewal fails before the expiry window.

Expected state: Renewal runs automatically with enough lead time and a tested fallback.

Decision points

Should the old certificate be revoked immediately?

Revoke immediately after the new certificate is live
Keep it valid until all clients migrate

Revoke immediately if the key was compromised; otherwise keep overlap only as long as needed for connection draining.

How much lead time should renewal have?

30 days before expiry
7 days before expiry

Use at least 30 days for public certificates so a failed renewal can be fixed without emergency work.

Expected outcome

The endpoint serves a valid replacement certificate, clients trust it, the old certificate is retired, and renewal is automated with an alerting path.

Verification

Use openssl or an external probe to confirm the served certificate's subject, SANs, issuer, and dates; confirm key-pinned clients and internal callers still connect; verify the old certificate is rejected or no longer served.

Cleanup

Remove temporary dual-certificate configuration, revoke compromised certificates, and update the inventory and runbook.

Inventory before expiry

Certificate outages are usually inventory failures. A team knows about the public certificate but forgets an internal service, a webhook endpoint, or a pinned mobile client. Start with a list of hostnames and owners, not a list of files on one server. Include the issuer, key type, expiry, and the automation that renews it.

openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

Run the check from outside the network as well as inside. An internal probe may trust a private CA that public clients do not.

Overlap, then drain

Install the new certificate alongside the old one where the server supports it, then reload gracefully. Existing keep-alive connections may continue using the old certificate until they close; that is acceptable during the overlap. Do not restart every instance at once. For a compromise, skip the overlap and revoke the old key immediately.

Automate with a dry run

ACME clients can renew automatically, but automation without monitoring just fails silently. Run certbot renew --dry-run in CI or on a schedule, alert on renewal failure, and track days-to-expiry as a metric. For certificates issued through a corporate CA, automate the request and deployment or document the manual steps with an owner.

Verify the chain, not just the date

A certificate can be unexpired and still fail because an intermediate is missing, a SAN is absent, or the key type is unsupported by an old client. Verify the full chain and the hostname, and test a representative client. Keep the old certificate available for rollback until the new chain is confirmed.