Retry Outbound Webhooks with Bounded Exponential Backoff

Deliver outbound webhooks reliably with bounded exponential backoff, jitter, Retry-After handling, and a replayable failure store for terminal errors.

~40 minwebhooksreliabilityqueues

Task

Add persistent retry scheduling with exponential backoff, jitter, and Retry-After support to an outbound webhook worker.

When to use this

A customer endpoint intermittently returns 503 or times out, and the current fire-and-forget sender drops the event after the first failed POST.

Prerequisites

Node.js 20 or later with PostgreSQL 14+ and Redis 7
BullMQ 5.x, ioredis 5.x, and node-postgres installed in the service
A webhook receiver that deduplicates events by delivery ID

Steps

1. Install the queue dependencies and start a local Redis instance for verification.

pnpm add bullmq ioredis pg
pnpm add -D tsx
docker run --rm -d --name webhook-redis -p 6379:6379 redis:7.4-alpine

Expected output: The runtime packages and tsx are added to package.json, and a Redis container named webhook-redis is running on port 6379.

2. Create a durable terminal-failure table so deliveries that exhaust retries survive queue cleanup.

psql "$DATABASE_URL" <<'SQL'
CREATE TABLE IF NOT EXISTS webhook_failures (
  delivery_id uuid PRIMARY KEY,
  endpoint_url text NOT NULL,
  event_type text NOT NULL,
  attempts integer NOT NULL,
  last_status integer,
  last_error text NOT NULL,
  payload jsonb NOT NULL,
  failed_at timestamptz NOT NULL DEFAULT now()
);
SQL

Expected output: psql prints CREATE TABLE and the webhook_failures table exists in the public schema.

3. Create the queue module with a typed job payload and a stable job ID for the first attempt.

import { Queue } from 'bullmq';
import IORedis from 'ioredis';

export const connection = new IORedis(process.env.REDIS_URL!, {
  maxRetriesPerRequest: null,
});

export type DeliveryJob = {
  deliveryId: string;
  endpointUrl: string;
  eventType: string;
  payload: unknown;
  attempt: number;
};

export const webhookQueue = new Queue<DeliveryJob>('webhooks', { connection });

export async function enqueueDelivery(input: Omit<DeliveryJob, 'attempt'>) {
  await webhookQueue.add(
    'deliver',
    { ...input, attempt: 1 },
    {
      jobId: input.deliveryId,
      removeOnComplete: 5000,
      removeOnFail: 5000,
    },
  );
}

Expected output: The first attempt is queued once per deliveryId; duplicate enqueues reuse the existing job ID.

4. Implement a worker that retries only transient outcomes, honors Retry-After, caps the delay, and writes exhausted deliveries to the failure table.

import { createHmac } from 'node:crypto';
import { Worker } from 'bullmq';
import pg from 'pg';
import { connection, webhookQueue, type DeliveryJob } from './queue.js';

const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL });
const MAX_ATTEMPTS = 8;
const BASE_DELAY_MS = Number(process.env.WEBHOOK_RETRY_BASE_MS ?? 30_000);
const MAX_DELAY_MS = Number(process.env.WEBHOOK_RETRY_MAX_MS ?? 21_600_000);

function retryAfterMs(header: string | null): number | null {
  if (!header) return null;
  const seconds = Number(header);
  if (Number.isFinite(seconds)) return Math.max(0, seconds * 1000);
  const at = Date.parse(header);
  return Number.isNaN(at) ? null : Math.max(0, at - Date.now());
}

function nextDelayMs(attempt: number, header: string | null): number {
  const exponential = Math.min(BASE_DELAY_MS * 2 ** (attempt - 1), MAX_DELAY_MS);
  const jittered = Math.floor(exponential * (0.5 + Math.random() * 0.5));
  const retryAfter = retryAfterMs(header) ?? 0;
  return Math.min(Math.max(jittered, retryAfter), MAX_DELAY_MS);
}

async function recordFailure(job: DeliveryJob, status: number | null, error: string) {
  await pool.query(
    `INSERT INTO webhook_failures
       (delivery_id, endpoint_url, event_type, attempts, last_status, last_error, payload)
     VALUES ($1, $2, $3, $4, $5, $6, $7)
     ON CONFLICT (delivery_id) DO UPDATE SET
       attempts = EXCLUDED.attempts,
       last_status = EXCLUDED.last_status,
       last_error = EXCLUDED.last_error,
       failed_at = now()`,
    [
      job.deliveryId,
      job.endpointUrl,
      job.eventType,
      job.attempt,
      status,
      error,
      JSON.stringify(job.payload),
    ],
  );
}

async function scheduleRetry(
  job: DeliveryJob,
  status: number | null,
  error: string,
  header: string | null,
) {
  if (job.attempt >= MAX_ATTEMPTS) {
    await recordFailure(job, status, error);
    return;
  }
  await webhookQueue.add(
    'deliver',
    { ...job, attempt: job.attempt + 1 },
    {
      delay: nextDelayMs(job.attempt, header),
      jobId: `${job.deliveryId}-${job.attempt + 1}`,
      removeOnComplete: 5000,
      removeOnFail: 5000,
    },
  );
}

export const webhookWorker = new Worker<DeliveryJob>(
  'webhooks',
  async (bullJob) => {
    const job = bullJob.data;
    const body = JSON.stringify(job.payload);
    const signature = createHmac('sha256', process.env.WEBHOOK_SECRET!)
      .update(body)
      .digest('hex');

    let response: Response;
    try {
      response = await fetch(job.endpointUrl, {
        method: 'POST',
        headers: {
          'content-type': 'application/json',
          'x-webhook-signature': signature,
          'x-webhook-delivery': job.deliveryId,
          'x-webhook-attempt': String(job.attempt),
        },
        body,
        signal: AbortSignal.timeout(10_000),
      });
    } catch (error) {
      const message = error instanceof Error ? error.message : 'network error';
      await scheduleRetry(job, null, message, null);
      return;
    }

    if (response.ok) return;

    const status = response.status;
    const retryable =
      status === 408 || status === 425 || status === 429 || status >= 500;

    if (!retryable) {
      await recordFailure(job, status, `permanent HTTP ${status}`);
      return;
    }

    await scheduleRetry(
      job,
      status,
      `transient HTTP ${status}`,
      response.headers.get('retry-after'),
    );
  },
  { connection, concurrency: 20 },
);

Expected output: A 503 response schedules the next attempt; a 400 response is recorded as permanent and is not retried.

5. Create a one-off enqueue script that accepts a fixed delivery ID so the verification query is deterministic.

import { randomUUID } from 'node:crypto';
import { connection, enqueueDelivery, webhookQueue } from './queue.js';

const deliveryId = process.env.DELIVERY_ID ?? randomUUID();
const endpointUrl = process.env.CUSTOMER_WEBHOOK_URL;

if (!endpointUrl) throw new Error('CUSTOMER_WEBHOOK_URL is required');

await enqueueDelivery({
  deliveryId,
  endpointUrl,
  eventType: 'order.created',
  payload: { orderId: 'ord_1042' },
});

console.log(deliveryId);
await webhookQueue.close();
await connection.quit();

Expected output: Running the script prints the delivery ID and exits 0 after the job is persisted in Redis.

6. Start a local receiver that fails twice and succeeds on the third attempt; leave it running and start the worker from step 4 in a second terminal with REDIS_URL=redis://127.0.0.1:6379 WEBHOOK_SECRET=local-test-secret WEBHOOK_RETRY_BASE_MS=500.

node --input-type=module -e "
import { createServer } from 'node:http';
let calls = 0;
createServer((req, res) => {
  calls += 1;
  console.log('receiver attempt', calls, req.headers['x-webhook-attempt']);
  if (calls < 3) {
    res.writeHead(503, { 'retry-after': '1' });
    res.end('retry');
    return;
  }
  res.writeHead(204);
  res.end();
}).listen(4100, () => console.log('receiver ready on 4100'));
"

Expected output: The receiver prints receiver ready on 4100 and waits for deliveries; the worker starts in the second terminal and connects to Redis.

7. With the worker running as REDIS_URL=redis://127.0.0.1:6379 WEBHOOK_SECRET=local-test-secret WEBHOOK_RETRY_BASE_MS=500 pnpm tsx src/webhooks/worker.ts, enqueue one delivery and confirm that no terminal failure row was written.

REDIS_URL=redis://127.0.0.1:6379 \
  DELIVERY_ID=11111111-1111-4111-8111-111111111111 \
  CUSTOMER_WEBHOOK_URL=http://localhost:4100/hooks/orders \
  pnpm tsx src/webhooks/enqueue.ts
psql "$DATABASE_URL" -c "SELECT count(*) AS failures FROM webhook_failures WHERE delivery_id = '11111111-1111-4111-8111-111111111111';"

Expected output: The receiver logs attempts 1, 2, and 3; the failure count is 0 because the third delivery returned 204.

Troubleshooting

Problem

The receiver processes the same event more than once

Solution

Retries are at-least-once by design. Deduplicate on the x-webhook-delivery ID inside one transaction and return 2xx for repeats.

Problem

A 429 endpoint stays backed up despite Retry-After

Solution

Retry-After is a lower bound, not a complete policy. Lower worker concurrency for that endpoint and cap the delay so one customer cannot starve others.

Problem

The worker cannot connect and BullMQ reports maxRetriesPerRequest errors

Solution

Set maxRetriesPerRequest to null on the shared ioredis connection and use a separate connection for blocking worker commands.

Limitations

Delivery is at least once, not exactly once; receiving systems must be idempotent.
Retries preserve no ordering between events after the first failure.
An endpoint that stays down longer than the maximum delay requires a manual replay from webhook_failures.

Retry by outcome, not by request

A webhook delivery is a state machine, not a single POST. Treat 2xx as acknowledged, retry 408, 425, 429, and 5xx, and stop immediately on most other 4xx responses because the request will never become valid. A 404 or 410 usually means the subscription is gone; a 401 means the receiver rejected the signature; retrying either just wastes capacity and delays the failure signal.

The schedule is exponential with a cap and jitter. If the first retry is 30 seconds, the next is 60, then 120, up to a six-hour ceiling. Jitter spreads a fleet of failed deliveries so they do not stampede a recovering endpoint at the same instant. Retry-After is treated as a lower bound: respecting a receiver’s request to slow down is both polite and effective, but the cap still prevents an absurd header from parking a job for a week.

Make failures durable

Queue retention is not an incident record. The webhook_failures table stores the payload, attempt count, last status, and error once a delivery becomes permanent or exhausts its budget. That gives support a replay path and gives monitoring a queryable signal. The worker recalculates the HMAC for every attempt because the signature must match the exact bytes sent, even though the payload is unchanged.

Verify the timeline

The local receiver returns 503 twice and 204 on the third call. With WEBHOOK_RETRY_BASE_MS=500 and Retry-After: 1, the retries complete in a few seconds, and the failure table stays empty. Then change the receiver to return 400 and confirm one row appears with attempts = 1. That second check proves permanent errors do not silently loop.

Pitfalls

Never retry indefinitely. A receiver that is down for a day needs a replay tool and an operator decision, not an unbounded queue. Do not put long-lived secrets in job data; load them by endpoint ID at send time. Use a deterministic delivery ID and tell customers to deduplicate, because the network can duplicate a request even when your scheduler does not. Finally, monitor the age of the oldest waiting job as well as the failure count; a growing delay queue is an outage that has not paged yet.

Example

import { Queue } from 'bullmq';
import IORedis from 'ioredis';

export const connection = new IORedis(process.env.REDIS_URL!, {
  maxRetriesPerRequest: null,
});

export type DeliveryJob = {
  deliveryId: string;
  endpointUrl: string;
  eventType: string;
  payload: unknown;
  attempt: number;
};

export const webhookQueue = new Queue<DeliveryJob>('webhooks', { connection });

export async function enqueueDelivery(input: Omit<DeliveryJob, 'attempt'>) {
  await webhookQueue.add(
    'deliver',
    { ...input, attempt: 1 },
    {
      jobId: input.deliveryId,
      removeOnComplete: 5000,
      removeOnFail: 5000,
    },
  );
}