Add a Request Priority Queue

Route interactive traffic ahead of batch work with separate queues, weighted workers, and aging so latency-sensitive requests keep their SLO.

~35 minqueuesredislatency

Task

Ensure background jobs cannot delay interactive API calls when the worker fleet is saturated.

When to use this

A single job queue mixes password-reset emails with nightly export generation. When a large export batch lands, password reset emails wait behind thousands of low-value jobs and users cannot sign in.

Prerequisites

Redis 7 or another queue with multiple lists or sorted sets
A worker service that can subscribe to more than one queue
Metrics for queue depth and oldest message age

Steps

1. Store jobs in two queues and pop the high-priority queue first with an atomic Lua script.

local job = redis.call('LPOP', KEYS[1])
if job then
  return {1, job}
end
job = redis.call('LPOP', KEYS[2])
if job then
  return {2, job}
end
return {0, ''}

Expected output: A worker always takes an interactive job before any batch job, with no race between two clients.

2. Assign most workers to interactive work but reserve capacity for batch so the low-priority queue still drains.

const WORKERS = 10;
const HIGH_WORKERS = 8;

for (let i = 0; i < WORKERS; i++) {
  const queues = i < HIGH_WORKERS
    ? ['jobs:interactive', 'jobs:batch']
    : ['jobs:batch'];
  startWorker(queues);
}

Expected output: Interactive traffic has eight dedicated workers; batch keeps two and cannot starve completely.

3. Track wait time per priority and alert when low-priority age exceeds its budget.

histogram.observe({ priority }, Date.now() / 1000 - job.enqueuedAt);

Expected output: Dashboards separate interactive and batch wait times, and batch age above 15 minutes pages the owner.

4. Test the ordering guarantee under load.

node scripts/enqueue-test.mjs --batch 10000 --interactive 1

Expected output: The single interactive job is processed in under one second despite 10,000 queued batch jobs.

Troubleshooting

Problem

Batch jobs never run during peak hours

Solution

Reserve at least one worker for the low-priority queue, or add aging that promotes jobs older than a threshold.

Problem

High-priority jobs still wait during a burst

Solution

Add admission control at the API edge and cap enqueue rate per tenant; priority changes ordering, not capacity.

Limitations

Priority does not create capacity; a saturated fleet still delays high-priority work.
Strict priority can starve low-priority work without reserved workers or aging.

Ordering is not capacity

A priority queue decides which job runs next, not how many jobs can run. If every worker is busy with a long export, a password reset still waits. Combine ordering with reserved capacity: most workers consume both queues with high priority first, and a minority consume only the low-priority queue. That guarantees forward progress for batch work while keeping interactive latency low.

Pop atomically

Two separate LPOP calls create a race where a high-priority job arrives between the two reads and waits for the next cycle. A small Lua script makes the check-and-pop atomic. Return the priority along with the payload so the worker can tag metrics and choose retry policy.

local job = redis.call('LPOP', KEYS[1])
if job then return {1, job} end
job = redis.call('LPOP', KEYS[2])
if job then return {2, job} end
return {0, ''}

Protect against starvation

Strict priority can starve batch work indefinitely. Reserve workers, or implement aging: when a batch job’s age exceeds a threshold, move it to the high-priority queue. Emit wait-time histograms per priority. A rising low-priority age with a healthy high-priority age is the signal that reserved capacity is too small.

Verify with a burst

Enqueue ten thousand batch jobs, then one interactive job, and measure how long the interactive job waits. It should complete in about one second. If it waits for the batch, the worker split is wrong or the pop is not atomic. Also test a worker crash: use a reliable queue pattern with BRPOPLPUSH or a visibility timeout so a job is not lost when a worker dies mid-flight.