Harden a systemd Service Against Crashes and Reboots

Set Restart, watchdog, and resource limits on a systemd unit so crashes self-heal, hangs get killed, and boot ordering is explicit.

~25 mindeploylinuxoperations

Task

Configure a systemd unit to survive crashes, detect hangs, restart with backoff, and start reliably after reboots.

When to use this

A worker process crashes at 3 a.m. and sits dead until someone notices at 9. When the server reboots for kernel patching, the service never comes back because it started before the network.

Prerequisites

A Linux host with systemd and root access
The service unit file under /etc/systemd/system
A test method to crash and hang the process

Steps

1. Enable automatic restart with bounded backoff.

[Service]
Type=exec
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=300
StartLimitBurst=5

Expected output: Crashes restart after five seconds, and more than five failures in five minutes stops retrying instead of thrashing forever.

2. Add a watchdog so hangs are detected.

[Service]
WatchdogSec=30

# in the application:
# sd_notify(0, "WATCHDOG=1");

Expected output: The process must ping systemd every thirty seconds; a hung process that stops pinging is killed and restarted automatically.

3. Declare what the service needs before starting.

[Unit]
After=network-online.target
Wants=network-online.target

[Install]
WantedBy=multi-user.target

Expected output: The unit waits for a usable network and is enabled at the multi-user target, so reboots bring it back reliably.

4. Bound its resources so it cannot take the host down.

[Service]
MemoryMax=1500M
LimitNOFILE=20000
CPUWeight=200

Expected output: A memory leak exhausts only its own cap and gets OOM-killed by systemd, not by a host-wide catastrophe.

5. Verify the whole lifecycle.

systemctl daemon-reload && systemctl enable --now worker && systemctl kill -s KILL worker && sleep 8 && systemctl is-active worker

Expected output: The service restarts within seconds of a hard kill, and enable guarantees the same behavior after reboot.

Troubleshooting

Problem

Service enters failed state after bursts

Solution

StartLimitBurst was hit by a real crash loop; investigate the root cause instead of raising the limit, which only hides it longer.

Problem

Watchdog kills a healthy service

Solution

The ping interval is too coarse for the watchdog window; ping more frequently than half of WatchdogSec, and never block the notify thread on I/O.

Problem

Fails to start after reboot but works manually

Solution

Ordering is implicit; add After and Wants for network and dependencies, and check early-boot logs with journalctl -b.

Limitations

Restart policies treat symptoms; a service restarting nightly is still broken and deserves a root-cause fix.
systemd watchdogs require application cooperation through sd_notify.

The babysitter you already have

systemd already contains the self-healing logic most people reimplement in wrapper scripts. Restart policies revive crashed processes with backoff, the watchdog kills processes that stop proving liveness, resource caps contain leaks, and unit ordering fixes boot-time races. Declaring these in the unit file replaces an unspecified pile of hope with a tested lifecycle contract.

[Service]
Type=exec
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=300
StartLimitBurst=5
WatchdogSec=30
MemoryMax=1500M

[Unit]
After=network-online.target
Wants=network-online.target

Backoff bounds are failure analytics

The start limit is not just protection against thrashing; it is an incident signal. Five failures in five minutes putting the unit into failed state means a crash loop got real, and the alert you attach to the failed state catches the loop the moment it forms rather than after a night of restart spam.

Watchdog closes the hang gap

Restart handles processes that die; the watchdog handles processes that stay alive but stop working. The application pings sd_notify on a healthy loop, and thirty silent seconds trigger a kill-and-restart. The two mechanisms together cover both failure modes a long-running service actually has.