Monitor Cron Jobs with Dead-Man's-Switch Heartbeats

1. Create a cron-type check with a two-hour grace period and capture the returned ping URL.

curl -sS -X POST https://healthchecks.io/api/v3/checks/ \
  -H "X-Api-Key: $HEALTHCHECKS_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"name":"nightly-billing-export","tags":"billing,cron","schedule":"0 2 * * *","tz":"UTC","grace":900}' \
  | jq '{name, schedule, grace, ping_url}'

Expected output: HTTP 201 with a JSON object containing name nightly-billing-export, schedule 0 2 * * *, grace 900, and a unique ping_url.

2. Create a backup script that pings only after pg_dump, a size check, and retention cleanup all succeed; a failure sends an explicit fail signal.

#!/usr/bin/env bash
set -euo pipefail

set -a
. /etc/billing-backup.env
set +a

PING_URL="${HEALTHCHECKS_PING_URL:?missing HEALTHCHECKS_PING_URL}"
OUT="/srv/backups/billing-$(date -u +%F).dump"

trap 'curl --fail --silent --max-time 10 "$PING_URL/fail" >/dev/null || true' ERR

pg_dump "$DATABASE_URL" --format=custom --no-owner --file="$OUT"
test -s "$OUT"
find /srv/backups -name 'billing-*.dump' -mtime +14 -delete

curl --fail --silent --show-error --max-time 10 --retry 3 "$PING_URL" >/dev/null

Expected output: A successful run creates a non-empty .dump file, deletes archives older than 14 days, and sends one success ping; any failed command sends /fail instead.

3. Install the script, write the two secrets into a root-only environment file, and add the cron entry without dropping existing jobs.

sudo install -m 0755 nightly-billing-backup.sh /usr/local/bin/nightly-billing-backup.sh
sudo install -m 0600 /dev/null /etc/billing-backup.env
sudo env DATABASE_URL="$DATABASE_URL" HEALTHCHECKS_PING_URL="$HEALTHCHECKS_PING_URL" \
  sh -c 'printf "DATABASE_URL=%s\nHEALTHCHECKS_PING_URL=%s\n" "$DATABASE_URL" "$HEALTHCHECKS_PING_URL" > /etc/billing-backup.env'

(sudo crontab -l 2>/dev/null; echo '0 2 * * * /usr/local/bin/nightly-billing-backup.sh >> /var/log/billing-backup.log 2>&1') | sudo crontab -
sudo crontab -l | grep nightly-billing-backup

Expected output: crontab prints the nightly-billing-backup line at 02:00, and /etc/billing-backup.env is mode 0600 with both values present.

4. Run the script once, inspect the archive, and use the fail and success endpoints to test the alert path without waiting for a missed schedule.

sudo /usr/local/bin/nightly-billing-backup.sh
latest=$(ls -t /srv/backups/billing-*.dump | head -1)
pg_restore --list "$latest" | head -5
curl -fsS "$HEALTHCHECKS_PING_URL/fail" >/dev/null
curl -fsS "$HEALTHCHECKS_PING_URL" >/dev/null

Expected output: pg_restore lists a valid archive, the /fail ping triggers the configured down alert, and the success ping resets the check.

5. Confirm the check status and last ping time through the Healthchecks API.

curl -sS -H "X-Api-Key: $HEALTHCHECKS_API_KEY" https://healthchecks.io/api/v3/checks/ \
  | jq -r '.checks[] | select(.name == "nightly-billing-export") | [.status, .last_ping] | @tsv'

Expected output: The row prints up followed by a recent ISO-8601 timestamp, and the dashboard shows the check in the up state.

Problem

The check alerts every day even though cron runs

Solution

Compare the cron timezone with the check timezone. Set CRON_TZ=UTC or change the check schedule; also verify that the job finishes inside the grace window.

Problem

No alert arrives when the job fails

Solution

Test /fail manually, confirm the integration is attached to this specific check, and verify the API key belongs to the same project.

Problem

The script works by hand but not from cron

Solution

Cron does not load the login profile. Source /etc/billing-backup.env from the script and use absolute paths for pg_dump, curl, and the backup directory.

Problem

The backup file exists but the heartbeat is missing

Solution

The final curl failed, likely because outbound HTTPS is blocked or the ping URL is wrong. With set -e, the job exits non-zero; check /var/log/billing-backup.log.