Prevent Overlapping Cron Runs

Guard a scheduled job with a lock or a run-state check so a slow run cannot overlap the next tick and double-process the same window.

~30 minschedulingreliabilityjobs

Task

Stop a scheduled job from running concurrently with a previous invocation of itself.

When to use this

A nightly job occasionally takes longer than its interval, so the next tick starts while the first still runs and both process the same records.

Prerequisites

A scheduler or cron runner
A shared store or the database for the lock

Steps

1. Record a run marker at start and clear it on successful completion.

Expected output: The next tick sees that a run is still active.

2. Skip execution when the marker is present and younger than the max expected runtime.

Expected output: Overlapping runs are prevented.

3. Use a stale-run reaper to clear markers older than the expected maximum.

Expected output: A crashed run does not block the job forever.

4. Make the job idempotent as a second line of defense.

psql -c "SELECT name, started_at FROM job_runs;"

Expected output: Even a forced run cannot double-apply changes.

Troubleshooting

Problem

Job permanently blocked by a crash

Solution

Add a reaper that clears markers older than the maximum reasonable runtime.

Problem

Clock drift across runners

Solution

Use the shared store clock for the marker rather than the runner local time.

Limitations

A lock prevents overlap but does not make the job incremental; a slow job may still skip a window if it cannot catch up.
It does not coordinate across schedulers that use different lock mechanisms.

Context

Stop a scheduled job from running concurrently with a previous invocation of itself. A nightly job occasionally takes longer than its interval, so the next tick starts while the first still runs and both process the same records.

Steps

Record a run marker at start and clear it on successful completion. Expected: The next tick sees that a run is still active.
Skip execution when the marker is present and younger than the max expected runtime. Expected: Overlapping runs are prevented.
Use a stale-run reaper to clear markers older than the expected maximum. Expected: A crashed run does not block the job forever.
Make the job idempotent as a second line of defense. Expected: Even a forced run cannot double-apply changes.

INSERT INTO job_runs (name, started_at)
VALUES ('nightly-rollup', now())
ON CONFLICT (name) DO NOTHING
RETURNING name;

Verify

The next tick sees that a run is still active.
Overlapping runs are prevented.
A crashed run does not block the job forever.
Even a forced run cannot double-apply changes.

Limits

A lock prevents overlap but does not make the job incremental; a slow job may still skip a window if it cannot catch up.
It does not coordinate across schedulers that use different lock mechanisms.