阅读中文版本

Build a Leased Distributed Task Scheduler

A complete scheduler that claims due tasks with leases, handles retries and dead letters, and survives worker crashes without double execution.

task-scheduler/
├── src/dispatch.ts
├── src/lease.ts
├── src/retry.ts
├── src/dead-letter.ts
├── tests/scheduler.test.ts
└── package.json

SELECT id FROM tasks
WHERE status = 'pending' AND next_run_at <= now()
ORDER BY next_run_at FOR UPDATE SKIP LOCKED LIMIT 10;

UPDATE tasks SET lease_owner = $1, lease_expires_at = now() + interval '60s'
WHERE id = ANY($2);

next_run_at = now() + (interval '1 second' * power(2, attempts));

UPDATE tasks SET status = 'dead_letter' WHERE id = $1 AND attempts >= max_attempts;

UPDATE tasks SET status = 'pending', lease_owner = NULL
WHERE status = 'running' AND lease_expires_at < now();

Project overview

Scheduled work is deceptively hard the moment there is more than one worker: two of them see the same due task, or one crashes mid-run and the task never runs again. This build makes exactly-once execution a property of the database rather than a hope.

You will store tasks with a next run time and a status, claim due rows with a locking select that two workers cannot both win, hold a lease while running, clear it on completion, and reclaim leases that outlive their worker.

The lease is the central idea. A worker that dies leaves its lease behind, and the lease expiry, not a heartbeat, is what lets another worker pick the task up, which avoids the failure mode where a crash makes a task invisible forever.

The scheduler keeps its workers deliberately stateless; all coordination lives in the task table. That is what lets you run more workers without a membership protocol or a leader election.

Each phase is useful alone: claiming, leasing, retry, dead-lettering, and reclamation, so an interrupted build still has a scheduler that runs tasks once.

The build is aimed at teams who reach for a cron container and discover, usually in production, that a single container is a single point of failure and that two containers run every task twice. It shows the smallest change that fixes both, and explains why the database, rather than a coordination service, is the natural place to enforce exactly-once execution.

What you will build

The scheduler delivers five capabilities, each with a concurrency or crash test:

• Claim due tasks
• Lease
• Retry with backoff
• Dead letter
• Reclaim expired leases

Claiming picks due tasks safely; a lease bounds how long a claim lasts; retry with backoff handles transient failure; a dead-letter state preserves work that cannot succeed; and reclamation returns leases abandoned by a crashed worker to the pool.

The design avoids a central scheduler process, because a single dispatcher is both a bottleneck and a single point of failure. Coordination through the database keeps the system flat and the failure modes local.

Architecture and data model

Tasks live in a table with a next_run_at, a status, and a lease. A dispatcher claims due rows with SELECT … FOR UPDATE SKIP LOCKED, sets a lease expiry, and runs them. Completion clears the lease and schedules the next run; repeated failures move to a dead-letter state.

A task row carries its schedule, its status, and a lease with an owner and an expiry. Workers claim due rows with a locking select that skips rows already claimed, then run them and update the row.

Completion clears the lease and computes the next run time; failure increments an attempt count and applies backoff. The same row therefore describes both the schedule and the execution history, which keeps reconciliation trivial.

Reclamation is a query: any lease past its expiry is treated as abandoned and returned to the claimable set. Expressing recovery as a query rather than a message is what makes a crashed worker a non-event.

Backoff is stored per task so a failing job does not retry at a fixed interval forever. Combining the attempt count with a cap turns a permanently broken task into a dead-lettered one rather than a hot loop.

The task row doubles as a lock through its lease, so the database’s own transaction boundaries provide mutual exclusion without a separate coordination service. That is why the correctness argument fits in a paragraph: a claim is a transactional update and a reclaim is a query over an expired lease, so there is no window in which two workers both believe they own the task.

Prerequisites and dependencies

You need a database that supports a locking select with row skipping, and workers that share a clock source.

• Node.js 20 LTS or later
• PostgreSQL 15+
• Two worker processes for the concurrency test

The dependencies keep scheduling, claiming, and persistence in one place:

• pg (^8.12.0) — Task table and row locking
• express (^4.21.0) — Status and enqueue endpoints

pnpm init
pnpm add pg express
pnpm add -D typescript tsx vitest

Use the database time for lease comparisons rather than the worker clock, or a worker with a skewed clock will either hoard tasks or release them too early. Record the clock source in configuration.

Directory structure

Separate the claim query, lease management, the retry policy, dead-letter handling, and reclamation so each is testable against a shared database:

task-scheduler/
├── src/dispatch.ts
├── src/lease.ts
├── src/retry.ts
├── src/dead-letter.ts
├── tests/scheduler.test.ts
└── package.json

Phase 1: Claim due tasks

Select due rows for update and skip locked ones.

SELECT id FROM tasks
WHERE status = 'pending' AND next_run_at <= now()
ORDER BY next_run_at FOR UPDATE SKIP LOCKED LIMIT 10;

SKIP LOCKED lets many workers claim disjoint tasks without blocking.

Claim due tasks with a single statement that selects for update and skips locked rows, so two workers running the same query cannot both receive the same task. This is the line that makes concurrency correct, and it should be tested with two workers.

Phase 2: Lease

Set a lease owner and expiry before running.

UPDATE tasks SET lease_owner = $1, lease_expires_at = now() + interval '60s'
WHERE id = ANY($2);

A lease bounds how long a crashed worker can hold a task.

Set a lease expiry when claiming, and renew it only if the work outlives the lease. A lease without an expiry is a task that can be lost silently when a worker dies holding it.

Phase 3: Retry with backoff

On failure increment attempts and schedule the next run with backoff.

next_run_at = now() + (interval '1 second' * power(2, attempts));

Transient failures retry automatically; permanent ones do not loop forever.

On failure, record the attempt and set the next run time from the backoff policy instead of retrying immediately. An immediate retry turns a brief downstream outage into a retry storm.

Phase 4: Dead letter

After the attempt cap, move the task to a dead-letter state.

UPDATE tasks SET status = 'dead_letter' WHERE id = $1 AND attempts >= max_attempts;

Poison tasks stop consuming worker time and become reviewable.

After a configured number of attempts, move the task to a dead-letter state and stop scheduling it. Preserving the payload and the last error lets an operator decide whether to fix and requeue or discard.

Phase 5: Reclaim expired leases

Return tasks whose lease expired to the pending state.

UPDATE tasks SET status = 'pending', lease_owner = NULL
WHERE status = 'running' AND lease_expires_at < now();

Crash recovery depends on lease reclamation.

Reclaim any lease past its expiry back into the due set, and log the reclaim with the previous owner. Reclamation is what turns a crashed worker from an incident into a routine event.

Testing

Run two dispatchers concurrently against the same database and assert each task ran once. Simulate a crash by leaving a lease and assert reclamation reschedules it.

Run two workers against the same database and assert that each due task ran exactly once. This test is the reason to choose a locking select over a simple read.

Simulate a crash by leaving a lease in place and advancing the clock past its expiry, then assert another worker reclaims and runs it. Without this test the recovery path is only an assumption.

Assert that a failing task backs off rather than retrying immediately, and that after the attempt limit it lands in dead-letter rather than looping.

Assert that a task scheduled in the future is not claimed, which guards the boundary that separates a scheduler from a naive queue.

Because claiming depends on the database’s locking semantics, the test must run against the same engine used in production; a suite against an in-memory database can pass while the real engine behaves differently under contention. Add the concurrent test early, because it is cheap and it protects the one property hardest to notice when it breaks.

pnpm vitest run

Deployment and operations

Run workers as stateless replicas behind any supervisor. Ensure clocks are close; lease logic depends on a shared notion of now, so use the database time or a synced NTP source.

Run workers as stateless replicas behind any supervisor, and size the fleet to the task arrival rate plus the worst-case task duration. Because coordination is in the database, adding a worker is safe at any time.

Keep clocks close by using the database time for lease logic; if workers are spread across regions, the lease comparison must not depend on the worker’s own clock.

Monitor claim contention and the lease expiry rate, because a rising expiry rate means tasks are running longer than their lease, which risks duplicate execution once recovery kicks in.

Alert on dead-letter growth, since a slow trickle of tasks that exhaust their attempts is usually one broken dependency rather than one broken task.

Leave headroom in the lease for the slowest observed task, and instrument the gap between task duration and lease duration so the margin never silently shrinks to zero. When that gap approaches zero, a slow task and a crash become indistinguishable, and the system starts reclaiming work that is still running.

Security considerations

Restrict task payloads to data, never code, and validate payloads before execution. Limit who can enqueue privileged task types.

Treat a task payload as untrusted if any part of it came from a user, because a scheduler that executes arbitrary payloads is a remote code path with a delay fuse.

Restrict which roles can insert into the task table, so one compromised service cannot schedule work on behalf of another.

Keep any credential a task needs scoped to the task’s purpose rather than to the worker, and never store secrets in the task row where an operator reading the table would see them.

Bound the number of tasks a single source can schedule, so a bug that creates tasks in a loop cannot fill the table and starve legitimate schedules.

Audit who can create a schedule, because a scheduler is a way to make the system do something later, which is exactly the capability an attacker wants after gaining a foothold.

Observability

Track pending count, oldest due task age, run duration, and dead-letter growth. Alert on a rising due-task backlog, which means workers are too few or too slow.

Track scheduled versus completed tasks over time and the lag between the two, because growing lag means the fleet is under-provisioned or a task is slower than assumed.

Chart the age of the oldest due-but-unclaimed task; a rising value is the earliest signal that no worker is picking up work, which is invisible in a completion rate alone.

Log the lease owner on claim and reclaim together with the task id, so a duplicate-execution investigation has a concrete timeline rather than a guess.

Alert when the reclamation rate spikes, since it usually means workers are crashing or tasks are overrunning their leases, and both degrade the exactly-once guarantee.

Publish the count of tasks in each state and the age of the oldest unclaimed due task, because a growing claimed set that never completes is the signature of workers dying without releasing their leases, and it is invisible in a completion rate alone.

Troubleshooting

A task runs twice. The handler is not idempotent and a lease expired mid-run; make effects repeatable or extend the lease.

Workers starve. The claim query lacked SKIP LOCKED and workers blocked on the same rows.

If a task runs twice, check the lease first: a lease shorter than the task duration lets a second worker reclaim while the first is still working.

If a task never runs, look for a stale lease that never expired because the status was left claimed; the reclaim query should be the only thing that decides whether a task is actually in flight.

If the same task appears to be pending forever with a claimed status, inspect its lease expiry and then the worker logs for that owner id, which narrows the fault to one process rather than the whole fleet.

Limitations and next steps

• The scheduler uses a database queue, not a dedicated broker.
• At-least-once semantics mean handlers must be idempotent.
• No cron-expression schedules; intervals and one-offs only.

The scheduler targets a single primary database; a multi-region deployment would need a global clock or a per-region partition, which is out of scope.

There is no dependency graph between tasks, so a task cannot declare that it must run after another completes, and ordering is left to the schedule itself.

Lease expiry is the only liveness signal; a long task must renew explicitly, and a task that forgets to renew will be reclaimed while still running.

The dead-letter store is a state rather than a queue with alerting workflows, so operators must poll it or export it to their existing tooling.

The scheduler keeps no history of completed runs beyond the last status, so auditing how often a task actually fired requires adding an append-only run log. It also assumes a single time source and a single primary, and a multi-region fleet would need clock discipline this example does not provide.