← Recipes

阅读中文版本

Alert Before a TLS Certificate Expires

Monitor cert-manager renewal status and expiry with a Prometheus rule, so a failed renewal pages someone before the outage.

~25 mintlscert-managermonitoring

Task

Detect a stuck certificate renewal days before expiry, not when browsers start failing.

When to use this

A DNS provider token expires, renewal silently fails, and the site goes down at midnight when the certificate expires; the alert fires only after users report errors.

Prerequisites

  • cert-manager with Prometheus scraping of its controller metrics
  • Alertmanager routed to the on-call channel
  • cmctl 2.0 or newer for manual inspection

Steps

1. Alert when any managed certificate is inside 21 days of expiry.

- alert: CertificateExpiringSoon
  expr: certmanager_certificate_expiration_timestamp_seconds - time() < 21 * 24 * 3600
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "Certificate {{ $labels.name }} expires in under 21 days"

Expected output: Every managed certificate warns well before the renewal window closes.

2. Alert on a certificate that cert-manager reports as not ready.

- alert: CertificateNotReady
  expr: certmanager_certificate_ready_status{condition="False"} == 1
  for: 30m
  labels:
    severity: critical
  annotations:
    summary: "Certificate {{ $labels.name }} in {{ $labels.namespace }} is not ready"

Expected output: A failed issuance or renewal pages immediately and names the certificate and namespace.

3. Inspect a specific certificate and request renewal when needed.

cmctl status certificate app-example-tls -n prod && cmctl renew app-example-tls -n prod

Expected output: Status shows the issuer, renewal time, and any challenge errors; renew starts a new issuance.

4. Prove the alert path with a short-lived test certificate in a scratch namespace.

Expected output: The warning lands in the alert channel with the correct name and namespace labels.

Troubleshooting

Problem

No certificate metrics are available

Solution

cert-manager must be scraped on its controller metrics port. Check the ServiceMonitor and confirm certmanager_* series exist.

Problem

The certificate renewed but the alert keeps firing

Solution

The ingress controller may still serve the old Secret. Check its logs and reload before silencing the alert.

Limitations

  • Metrics disappear if cert-manager itself is down, so pair this with an external TLS probe that checks the served chain.
  • Very short-lived certificates need a percentage-based threshold instead of a fixed 21-day window.

Expiry is a deadline you can see coming

TLS expiry is one of the few outages with a known date, which makes it inexcusable. cert-manager already tracks the expiry timestamp and readiness of every Certificate it manages; the missing piece is an alert that fires while there is still time to act. A 21-day warning catches a renewal that failed on the first attempt, and a readiness alert catches a failure early enough to investigate the cause. Together they turn a midnight outage into a ticket with a week of runway.

Alert on two different conditions

Expiry and readiness fail differently. A certificate can be ready today and still expire in twenty days because renewal never succeeded, and a renewal can fail today on a certificate with sixty days left. Alerting only on expiry misses the cause; alerting only on readiness can miss a certificate that is not managed by cert-manager at all. Keep both rules, and route the readiness alert at critical because it means an automated process is already broken.

- alert: CertificateExpiringSoon
  expr: certmanager_certificate_expiration_timestamp_seconds - time() < 21 * 24 * 3600
  for: 1h
  labels:
    severity: warning

Verify from the outside too

Metrics describe what cert-manager believes; clients see what the ingress actually serves. Add an external probe that reads the certificate chain for each public hostname and alerts when expiry is close, because that probe catches the case where renewal succeeded but the new Secret never reached the data plane. Test the whole path by issuing a deliberately short-lived certificate in a scratch namespace and confirming the alert, the routing, and the runbook all work. Then rotate the ACME account key and DNS credentials on a schedule, since those are the usual cause of silent renewal failure.