Guarantee Cleanup Runs When a Shell Script Exits

1. Create resources in trackable variables and install the trap immediately.

#!/usr/bin/env bash
set -euo pipefail

LOCKFILE=/tmp/app.lock
WORKDIR=""; CHILD_PIDS=()

cleanup() {
  # kill children first (they may hold the workdir)
  if ((${#CHILD_PIDS[@]})); then
    kill "${CHILD_PIDS[@]}" 2>/dev/null || true
    wait "${CHILD_PIDS[@]}" 2>/dev/null || true
  fi
  [ -n "$WORKDIR" ] && rm -rf "$WORKDIR"
  [ -f "$LOCKFILE" ] && rm -f "$LOCKFILE"
}
trap cleanup EXIT
trap 'exit 130' INT TERM   # convert signals into an EXIT

# now acquire resources
exec 9>"$LOCKFILE"
flock -n 9 || { echo "already running" >&2; exit 1; }
WORKDIR=$(mktemp -d /tmp/app.XXXXXX)

Expected output: Whatever exit path the script takes, the same cleanup runs: children reaped, workdir removed, lock released.

2. Register child processes as they spawn.

run_step() {
  "$@" &
  CHILD_PIDS+=($!)
  wait $!
}

run_step ./etl-extract --to "$WORKDIR/raw"
run_step ./etl-transform --in "$WORKDIR/raw" --out "$WORKDIR/clean"
run_step ./etl-load --from "$WORKDIR/clean"

Expected output: Every background process is in the registry, so an interrupt during step three still kills extract and transform leftovers.

3. Prove the interrupt path actually cleans up.

bash job.sh & JOB=$!; sleep 2; kill -INT $JOB; wait $JOB; ls /tmp/app.lock 2>/dev/null && echo "LEAK: lock survived" || echo "lock released"; ls -d /tmp/app.* 2>/dev/null | grep -v lock || echo "workdir removed"

Expected output: The test prints lock released and workdir removed: the trap demonstrably fires on SIGINT, not only on a clean exit.

4. Keep the trap idempotent and cheap for double-fire safety.

CLEANED=0
cleanup() {
  [ "$CLEANED" -eq 1 ] && return 0
  CLEANED=1
  trap - EXIT INT TERM      # disarm before acting
  # ... actual cleanup ...
}
trap cleanup EXIT
trap 'cleanup; exit 143' TERM

Expected output: A signal arriving during cleanup cannot re-enter the function, and disarmed traps prevent recursion while rm -rf runs.

5. Test the error path too, not just Ctrl-C.

bash -c 'source job.sh; false' 2>/dev/null; ls /tmp/app.lock 2>/dev/null || echo "error path: clean"; bash job.sh --inject-failure 2>/dev/null; ls -d /tmp/app.* 2>/dev/null | wc -l

Expected output: Both an early command failure and a mid-run injected failure leave zero stray files, covering the set -e exit path as well.

Problem

Cleanup runs but children survive

Solution

wait after kill, and start children in their own process group with setsid so a terminal-wide signal cannot be outrun; plain kill on the script does not propagate to grandchildren by itself.

Problem

trap on EXIT fires twice

Solution

Signal traps that call exit re-enter the EXIT trap; disarm traps inside cleanup (trap - EXIT INT TERM) and guard with a CLEANED flag as the double-fire latch.

Problem

flock releases but a stale lock blocks the next run

Solution

You are mixing flock with a manual lockfile; pick one: flock releases automatically when fd 9 closes on any exit, while manual lockfiles need the trap plus a staleness check on PID.