阅读中文版本

Build an Async Job Priority Queue

A multi-tier async job queue with priority lanes, fair scheduling across lanes, delayed dispatch, and per-worker concurrency control.

async-job-priority-queue/
├── src/coordinator.ts
├── src/picker.ts
├── src/worker.ts
├── src/retry.ts
├── tests/queue.test.ts
└── package.json

const lanes = ["high","default","low"]; const weights = {high:6,default:3,low:1};

await rpush(lane, job); if (delay) await zadd("delayed", now+delay, job.id);

const job = await pull(workerId); await process(job); await ack(job.id);

if (job.attempts >= MAX) await lpush("dead", job); else await requeue(job);

Project overview

A priority queue that only ever serves the highest lane is not a priority queue, it is a monopoly. The moment a burst of high-priority work arrives, every other lane stops, and the system that was supposed to be fair becomes a funnel. This build keeps high-priority work fast without granting it exclusive access to the workers.

The coordinator holds three FIFO lanes and drains them in a weighted round-robin: six pulls from high, three from default, one from low. Under a sustained high-priority load, low-priority jobs still move, just slowly. Under normal load, high-priority jobs jump the queue because the picker prefers a non-empty high lane.

The design treats workers as unreliable. A worker claims a job and must ack within a heartbeat window; if it crashes, the coordinator re-queues the job after two missed beats. This makes the queue safe to scale horizontally, because losing a worker never loses work.

The retry path is bounded. A job that fails increments an attempt counter, and at the cap it moves to a dead-letter list rather than looping forever. The dead-letter keeps the original payload and the failure trace, so an operator can inspect what went wrong without re-running the job.

The value of this project is the weighting, not the queueing. A FIFO is trivial; a fair-share picker that degrades gracefully under one-sided load is the part worth learning, because it is the difference between a queue that serves the system and one that serves only the loudest caller.

What you will build

The project delivers five capabilities, each driven by a test:

• Three priority lanes
• Fair-share across lanes
• Delayed dispatch
• Per-worker concurrency cap
• Dead-letter on max retries

A coordinator holds three FIFO lanes (high, default, low). A weighted round-robin picker drains lanes in a 6:3:1 ratio so low-priority jobs still progress under load. Workers pull via long-poll; the coordinator caps each worker to a configurable concurrency and re-queues orphaned jobs on heartbeat timeout.

Architecture and data model

The coordinator is the single source of truth for lane state, backed by Redis so a coordinator restart does not lose in-flight jobs. Three FIFO lists hold ready jobs, a sorted set holds delayed jobs keyed by due time, and a hash holds worker heartbeats.

The picker is a weighted round-robin that tracks how many consecutive pulls it has taken from each lane. When a lane is empty it is skipped, so the weights are a ceiling on attention, not a floor. A lane with no work costs nothing.

Workers pull via a long-poll that blocks up to a configurable timeout, so an idle worker does not hammer Redis. On claim, the job moves to an in-flight hash keyed by job id with the worker id and a deadline. A sweeper scans the in-flight hash periodically and re-queues any job whose deadline has passed.

The delay index is a sorted set of job ids scored by due time. A ticker runs every second, moves due jobs from the sorted set to their target lane, and removes them from the delay index. This separates scheduling from dispatch, so a job can be enqueued for the future without occupying a lane.

The dead-letter list is a plain Redis list. Jobs land there at max retries and stay until an operator inspects or purges. The queue does not auto-retry from dead-letter, because silent retries of a job that has repeatedly failed are how a system accumulates hidden damage.

Prerequisites and setup

You need the following before you begin:

• Node.js 20 LTS or later
• Redis 7+ for the queue backing store
• At least one worker process

The dependencies cover the core needs:

• ioredis (^5.4.0) — Queue backing store
• pino (^9.4.0) — Structured logs
• zod (^3.23.0) — Job payload validation

pnpm init
pnpm add ioredis
pnpm add pino
pnpm add zod
pnpm add -D typescript tsx vitest

Keep secrets and configuration in the environment, and validate them at boot so a misconfigured service fails loudly instead of silently dropping work.

Directory structure

async-job-priority-queue/ is organized so each concern is testable in isolation:

async-job-priority-queue/
├── src/coordinator.ts
├── src/picker.ts
├── src/worker.ts
├── src/retry.ts
├── tests/queue.test.ts
└── package.json

Phase 1: Lane model

Define three FIFO lanes and a weighted picker that drains them in a 6:3:1 ratio.

const lanes = ["high","default","low"]; const weights = {high:6,default:3,low:1};

Picker returns the next lane by weight, falling back to any non-empty lane.

The phase is driven by a test that asserts the expected behavior, so the implementation is correct before the next phase builds on it.

Phase 2: Enqueue with delay

Push a job onto a lane, optionally scheduling it for future dispatch via a sorted-set delay index.

await rpush(lane, job); if (delay) await zadd("delayed", now+delay, job.id);

A ticker moves due delayed jobs into their target lane.

The phase is driven by a test that asserts the expected behavior, so the implementation is correct before the next phase builds on it.

Phase 3: Worker pull and ack

A worker long-polls the coordinator, claims a job, and must ack within a heartbeat window or the job is re-queued.

const job = await pull(workerId); await process(job); await ack(job.id);

Unacked jobs are re-queued after two missed heartbeats.

The phase is driven by a test that asserts the expected behavior, so the implementation is correct before the next phase builds on it.

Phase 4: Retry and dead-letter

On failure, increment a retry counter; at max retries move the job to a dead-letter list for inspection.

if (job.attempts >= MAX) await lpush("dead", job); else await requeue(job);

Dead-letter jobs keep their original payload and failure trace.

The phase is driven by a test that asserts the expected behavior, so the implementation is correct before the next phase builds on it.

Testing

Enqueue a high and a low job, assert high dispatches first; flood the default lane and assert low still progresses at the weighted ratio; kill a worker mid-job and assert the job is re-queued after heartbeat timeout.

The tests drive the real coordinator against an in-memory Redis shim, so they run fast without a network. The fairness test floods the default lane with a hundred jobs, enqueues one low-priority job, and asserts the low job dispatches within the weighted window, proving the picker does not starve.

The heartbeat test claims a job, stops acking, and asserts the job re-queues after the deadline plus one sweep. The delay test enqueues a job with a two-second delay and asserts it does not appear in any lane before the due time, then appears after.

The dead-letter test fails a job repeatedly and asserts it lands in the dead-letter list at the cap, not on the ready lane. The ordering test enqueues one job per lane and asserts the high job dispatches first, the default second, the low third, under no other load.

The concurrency-cap test starts a worker with a limit of two, claims two jobs, and asserts a third pull returns nothing until one of the first two is acked. This pins the per-worker cap as a tested invariant.

pnpm vitest run

Observability

Track the three priority lanes rate, the queue resource usage, and the error count as the three primary signals for the async-job-priority-queue system. A sudden change in any of these is the signal that something has shifted, not necessarily broken but worth a look before it becomes an incident. Chart the fair-share across lanes alongside the three priority lanes so a correlation is visible at a glance, because a rising fair-share across lanes with a stable three priority lanes means the system is doing more work per unit, not that it is failing.

Log every lane model transition with the context that triggered it, because an operator who sees the symptom but not the cause will guess wrong. The log entry should carry the queue identifier, the input that caused the transition, and the outcome, so the operator can reconstruct what happened without re-running it. A log without context is a line of text; a log with context is a diagnosis.

Set alerts on the derived metrics, not the raw counters. A rate of change is more actionable than a total, because a total grows monotonically and never tells you when the growth accelerated. Alert on the three priority lanes error rate crossing a threshold, the queue utilization exceeding a percentage, and the fair-share across lanes latency exceeding the SLO. Each alert should have a runbook link, so the operator who receives it at three in the morning knows what to do, not just what is wrong.

The async-job-priority-queue system should expose its metrics in a format the existing dashboard can scrape, so the observability is additive, not a new tool. A metric that nobody reads is noise, so before adding one, confirm it has a consumer. The dashboard has three panels: throughput, error rate, and resource utilization, each with a threshold line that is the SLO, so a glance tells an operator whether the system is healthy without reading numbers.

Deployment and operations

Run the coordinator as a singleton and workers as a pool behind a process manager that restarts on crash. Point every process at the same Redis, and give each worker a unique id so the in-flight hash does not collide.

Scale workers per lane if one priority dominates the traffic, by running a worker pool that only pulls from the high lane and another that pulls from all. This gives high-priority work dedicated capacity without changing the picker.

Monitor lane depth, in-flight count, and dead-letter growth. A growing dead-letter list means a downstream is failing; a growing lane depth means workers are saturated or the picker is mis-weighted. Both are operator signals, not code bugs.

Set the heartbeat window from observed processing time, not a guess. Too short and slow jobs are re-queued and run twice; too long and a crashed worker blocks its jobs before they recover. The window belongs in configuration and in review.

Security considerations

Treat every input to the async-job-priority-queue system as untrusted, because the three priority lanes path is the one place an external caller controls what the system processes. Validate the input shape before any queue operation, and reject early with a clear error rather than letting a malformed input propagate into the lane model phase where it causes a confusing failure that is hard to trace back to its source.

Keep secrets and configuration in the environment, never in the repository. The ioredis credentials, the signing keys, and the service endpoints are all environment variables, read once at boot and validated, so a misconfigured async-job-priority-queue fails loudly at startup instead of silently misbehaving at runtime. A system that starts with a bad configuration and runs anyway is a system that will produce wrong results for hours before anyone notices.

Rate-limit the three priority lanes endpoint per caller, because an unbounded queue operation is a resource leak as much as a security problem. A caller that can trigger the enqueue with delay path without limit can exhaust the system, whether intentionally or not. The rate limit is a per-caller ceiling that costs legitimate callers nothing and stops the abuse before it reaches the lane model logic.

Audit-log every state-changing operation with the caller identity, the input, and the timestamp. The audit log is not for debugging, it is for accountability: if something went wrong, the log says who did what and when. The async-job-priority-queue system writes the audit log to an append-only store that the system itself cannot modify, so a compromised system cannot rewrite its history.

Do not expose internal errors to the caller. A stack trace is a map of the system internals, and an attacker who collects them learns the queue structure. Return a generic error with a correlation id, and log the full error internally with the same id, so an operator can trace it without leaking it.

Limitations and next steps

• Single Redis instance is a bottleneck and a SPOF; shard by lane for higher throughput.
• No job deduplication; identical payloads enqueue independently.
• Weights are static; adaptive weighting is out of scope.

A single Redis is both the bottleneck and the single point of failure. For higher throughput, shard by lane across multiple Redis instances, so the high lane gets a dedicated connection pool. For resilience, run Redis in sentinel or cluster mode, though the coordinator does not require cluster-specific commands.

There is no job deduplication. Two identical payloads enqueue independently and run twice. If that is a problem, dedup belongs in the producer, not the queue, because the queue cannot know what makes two jobs identical.

The weights are static. A lane that is quiet at night and loud at noon uses the same ratio, which is fine until traffic shifts and the ratio no longer matches. Adaptive weighting, where the picker learns from lane depth, is a real improvement but out of scope here.