Automate Certificate Renewal with a Deploy Hook

Configure certbot renewals with dry runs, a deploy hook that reloads the right service, and monitoring that alerts when renewal stops happening.

~20 mindeploytlsautomation

Task

Make TLS certificate renewal fully automatic including service reload, and prove the chain works before the real expiry date.

When to use this

The certificate expired at the worst possible moment because the systemd timer was disabled during a host migration. Nobody noticed until browsers showed warnings.

Prerequisites

A host with certbot installed and a certificate issued
The consuming service, such as nginx, reloadable without downtime
Root access to configure timers and hooks

Steps

1. Check the renewal configuration exists and is valid.

certbot renew --dry-run

Expected output: The dry run validates ACME account, challenge type, and authorization end to end without touching the live certificate.

2. Add a deploy hook that reloads the consumer.

#!/bin/sh
# /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh
nginx -t && nginx -s reload

Expected output: The hook runs only when a certificate actually renews, reloading nginx gracefully with a config check first.

3. Confirm the timer is enabled and scheduled.

systemctl list-timers | grep certbot && systemctl status certbot.timer | head -3

Expected output: The timer shows a next-run time, proving renewals will fire twice daily as packaged.

4. Test the full chain including the hook.

certbot renew --dry-run --force-renewal 2>/dev/null; ls -la /etc/letsencrypt/renewal-hooks/deploy/

Expected output: Dry-run output and hook permissions confirm the chain would execute correctly on a real renewal.

5. Monitor the certificate lifetime so silence alerts.

- name: Alert on short certificate lifetime
  run: |
    days=$(openssl x509 -enddate -noout -in /etc/letsencrypt/live/example.com/fullchain.pem \
      | cut -d= -f2 | date -d "$(cat)" +%s 2>/dev/null; echo $(( (days - $(date +%s)) / 86400 )))
    test "$days" -gt 14 || { echo "cert expires in $days days"; exit 1; }

Expected output: A daily check fails loudly when the certificate has fewer than fourteen days left, catching any silently broken renewal chain.

Troubleshooting

Problem

Renewal succeeds but the site serves the old certificate

Solution

The service was never reloaded; the deploy hook is the fix, and it must restart or reload the exact consumers of that certificate file.

Problem

Challenge fails after a firewall change

Solution

The ACME challenge port, 80 or 443 for HTTP-01, was blocked; allow it or switch to a DNS-01 challenge that needs no inbound path.

Problem

Multiple services share one certificate

Solution

List every consumer in the deploy hook; a hook that reloads nginx but not haproxy leaves half the fleet stale.

Limitations

Dry runs validate the chain but not the final reload behavior on the live traffic path; schedule one real renewal observation.
ACME rate limits make force-renewal rehearsals expensive; rely on dry-run plus monitoring instead.

Renewal is not done when certbot exits

A successful renewal writes new files; it does not reload nginx, does not survive a disabled timer, and does not announce its own failure. The complete automation has three parts: the renewal itself on a verified timer, a deploy hook that reloads every consumer of the certificate, and an independent monitor on the remaining lifetime that screams when any link in that chain silently breaks.

# The deploy hook
#!/bin/sh
nginx -t && nginx -s reload

# The independent watchdog
openssl x509 -enddate -noout -in /etc/letsencrypt/live/example.com/fullchain.pem
# alert when fewer than 14 days remain

The hook is per-consumer, not per-host

One certificate often feeds several services: nginx, haproxy, an SMTP daemon. The deploy hook must reload all of them, or renewal silently leaves half the fleet serving the expired file. Inventory the consumers once, encode them in the hook, and the inventory lives where the next engineer will find it.

Watch the outcome, not the mechanism

Monitoring certbot timer status tells you the mechanism ran; monitoring the certificate end date tells you the outcome is still fine. The second is the one that matters, because it catches every failure mode including disabled timers, broken hooks, and failed challenges, with a single check and no coupling to certbot internals.