阅读中文版本

Build a Webhook Forwarder with Signed Retries

A webhook relay that queues inbound events, delivers them downstream with HMAC signatures, retries with exponential backoff, and exposes delivery status per event.

webhook-forwarder/
├── src/
│   ├── server.js
│   ├── dispatcher.js
│   ├── signer.js
│   ├── queue.js
│   └── config.js
├── test/
│   ├── signer.test.js
│   └── dispatcher.test.js
├── hooks.json
└── package.json

const crypto = require('node:crypto');

function sign(secret, bodyBuffer, timestamp = Math.floor(Date.now() / 1000)) {
  const mac = crypto.createHmac('sha256', secret);
  mac.update(timestamp + '.');
  mac.update(bodyBuffer);
  return {
    'x-hook-timestamp': String(timestamp),
    'x-hook-signature': 'sha256=' + mac.digest('hex'),
  };
}

const claimDue = db.prepare(`
    UPDATE deliveries SET status = 'in-flight', attempts = attempts + 1
    WHERE id = (SELECT id FROM deliveries
                WHERE status = 'queued' AND next_attempt_at <= ?
                ORDER BY next_attempt_at LIMIT 1)
    RETURNING *`);
const sweepStale = db.prepare(`
    UPDATE deliveries SET status = 'queued', next_attempt_at = ?
    WHERE status = 'in-flight' AND claimed_at < ?`);

async function deliverOne(hook, event) {
  const res = await fetch(hook.url, {
    method: 'POST',
    headers: { 'content-type': 'application/json', ...sign(hook.secret, event.body) },
    body: event.body,
  });
  if (res.ok) return { ok: true };
  return { ok: false, retryable: res.status >= 500 || res.status === 429, status: res.status };
}

app.post('/hooks/:name', (req, res) => {
  const chunks = [];
  req.on('data', (c) => chunks.push(c));
  req.on('end', () => {
    const body = Buffer.concat(chunks);
    const id = queue.enqueue(req.params.name, body);
    res.writeHead(202, { location: '/hooks/' + req.params.name + '/deliveries/' + id });
    res.end();
  });
});

Project overview

Webhooks are where reliability engineering gets concrete: networks drop, receivers 500, and clocks drift. A forwarder that simply POSTs and hopes produces silent data loss. This project builds the boring machinery that makes delivery honest: a durable queue, a signed envelope, and a retry policy with a terminal state.

The design target is at-least-once delivery with verifiable authenticity. Exactly-once is impossible over plain HTTP without receiver cooperation, so the project states the contract, signs every payload, and gives receivers the event identity they need to deduplicate.

The finished service delivers five capabilities:

• POST /hooks/:name accepts an event and returns 202 with a queue position and a location header.
• Signed delivery — every outbound POST carries an HMAC-SHA256 signature and timestamp header computed over the exact body bytes.
• Retries with capped exponential backoff and per-endpoint attempt limits, honoring 429 Retry-After when present.
• GET /hooks/:name/deliveries lists status, attempts, last error, and next retry time per event.
• A dead letter status marks events that exhausted retries, kept queryable for replay.

The architecture is three stages over one durable spine: an HTTP ingest layer validates and enqueues events into SQLite, a dispatcher loop claims due deliveries one at a time, and a signer module computes the HMAC envelope. Separating claim from delivery means a crash mid-delivery leaves the row marked in-flight and a sweeper requeues it — at-least-once semantics with idempotency delegated to the receiver, which is the only honest division of labor over HTTP.

Setup

Verify the prerequisites before starting: Node.js 20 LTS or later, an understanding of HMAC and why raw body bytes matter for signatures, and familiarity with SQLite or any embedded database.

The dependency footprint is one runtime package plus the test runner:

Create the project and install:

mkdir webhook-forwarder && cd webhook-forwarder
npm init -y
npm install better-sqlite3@^11.0.0
npm install -D vitest@^2.1.0

Configure the environment in package.json — the server entry, the test script, and the queue file plus hook definitions from the working directory, with per-hook secrets through environment files rather than hooks.json:

{
  "name": "webhook-forwarder",
  "scripts": { "start": "node src/server.js", "test": "vitest run" }
}

The directory structure mirrors the stages: signer.js is pure crypto over buffers, queue.js owns SQLite and the claim/sweep queries, dispatcher.js owns the retry loop, and server.js owns HTTP.

Phase 1: Signed envelope

The signer computes an HMAC-SHA256 over the exact request body bytes with a per-hook secret and produces the signature headers.

// src/signer.js
import { createHmac } from 'node:crypto';

export function sign(secret, bodyBuffer, timestamp = Math.floor(Date.now() / 1000)) {
  const mac = createHmac('sha256', secret);
  mac.update(timestamp + '.');
  mac.update(bodyBuffer);
  return {
    'x-hook-timestamp': String(timestamp),
    'x-hook-signature': 'sha256=' + mac.digest('hex'),
  };
}

The envelope design is the phase’s whole content. The digest covers the timestamp, a dot, and the exact body bytes — receivers recompute the digest with the shared secret, reject timestamp skew beyond a few minutes to blunt replay, and only then parse the body. The timestamp prefix means an intercepted payload cannot be replayed next week without breaking the freshness window, and it is signed so a receiver can trust the clock it is checking.

Signing bytes rather than a re-serialized object is the entire trick: JSON.parse followed by JSON.stringify can reorder keys, drop or normalize whitespace, and silently invalidate the signature — which is why the ingest layer in phase 4 persists raw body buffers and the signer receives exactly those bytes. The test suite pins the digest against a vector computed with the Node crypto shell, so a regression in the envelope is caught as a hex mismatch rather than a mystery 401 downstream, and a one-byte mutation test confirms the digest actually depends on the payload.

Acceptance for this phase: signer tests verify the digest against a precomputed vector and confirm that mutating one body byte breaks the signature.

Phase 2: Durable queue

The queue module creates the deliveries table with status, attempts, next_attempt_at, and an in-flight marker, plus the claim and sweep queries the dispatcher uses.

// src/queue.js
import Database from 'better-sqlite3';

export function createQueue(dbFile) {
  const db = new Database(dbFile);
  db.pragma('journal_mode = WAL');
  db.exec(`
    CREATE TABLE IF NOT EXISTS deliveries (
      id INTEGER PRIMARY KEY,
      hook TEXT NOT NULL,
      body BLOB NOT NULL,
      status TEXT NOT NULL DEFAULT 'queued',
      attempts INTEGER NOT NULL DEFAULT 0,
      next_attempt_at INTEGER NOT NULL,
      claimed_at INTEGER,
      last_error TEXT
    );
  `);

  const claimDue = db.prepare(`
    UPDATE deliveries SET status = 'in-flight', attempts = attempts + 1,
        claimed_at = ?
    WHERE id = (SELECT id FROM deliveries
                WHERE status = 'queued' AND next_attempt_at <= ?
                ORDER BY next_attempt_at LIMIT 1)
    RETURNING *`);

  const sweepStale = db.prepare(`
    UPDATE deliveries SET status = 'queued', next_attempt_at = ?
    WHERE status = 'in-flight' AND claimed_at < ?`);

  return { db, claimDue, sweepStale, enqueue: /* ... */ enqueueImpl(db) };
}

The claim query is one atomic statement — an UPDATE ... RETURNING that flips the row to in-flight and increments attempts in the same operation the select found it. Exactly-one claiming falls out of the atomicity: a second immediate claim returns no rows, because the first claim’s status flip is already committed, and the queue tests assert that property directly rather than hoping it.

The sweeper is the crash-recovery story: a delivery marked in-flight whose claimed_at is older than the crash threshold gets requeued by the sweep, so a process kill between claim and delivery strands the row for at most one threshold interval instead of forever. The body is stored as a BLOB — the raw bytes — because phase 1’s signature contract requires delivering exactly what was received, not a re-serialization.

Acceptance for this phase: claim returns exactly one due delivery and a second immediate claim returns none; sweeping requeues a delivery marked in-flight before the crash threshold.

Phase 3: Dispatcher with backoff

The dispatcher runs the claim-deliver-record loop, applying capped exponential backoff on failure and marking dead letter once attempts are exhausted.

// src/dispatcher.js
export function createDispatcher({ queue, hooks, sign, fetchImpl = fetch,
                                   now = () => Date.now(),
                                   backoff = { base: 1000, cap: 60000, maxAttempts: 8 } }) {
  async function tick() {
    const event = queue.claimDue.get(now(), now());
    if (!event) return;
    const hook = hooks[event.hook];

    let result;
    try {
      const res = await fetchImpl(hook.url, {
        method: 'POST',
        headers: { 'content-type': 'application/json', ...sign(hook.secret, event.body) },
        body: event.body,
      });
      result = res.ok
        ? { ok: true }
        : { ok: false, retryable: res.status >= 500 || res.status === 429, status: res.status };
    } catch (err) {
      result = { ok: false, retryable: true, error: err.message };   // network errors retry
    }
    record(event, result);
  }

  function record(event, result) {
    if (result.ok) return queue.markDelivered(event.id);
    if (!result.retryable || event.attempts >= backoff.maxAttempts) {
      return queue.markDead(event.id, result.error ?? 'status ' + result.status);
    }
    const delay = Math.min(backoff.cap, backoff.base * 2 ** (event.attempts - 1));
    queue.scheduleRetry(event.id, now() + delay, result.error ?? result.status);
  }

  return { tick };
}

Failure classification drives the retry policy: 5xx and 429 are retryable with capped exponential backoff, while 4xx responses mark the event dead immediately, because no amount of retrying fixes a request the receiver considers malformed — and retrying it anyway is how a struggling receiver gets hammered. Network exceptions (connection refused, DNS failure, timeout) are retryable, same as a 5xx, and the distinction is explicit in the code rather than accidental.

The backoff curve is base * 2^(attempt-1) capped at cap, and a 429 carrying Retry-After is honored as authoritative when present, overriding the computed delay. Dead letter rows stay queryable for replay after the receiver fixes their bug — terminal is a status, not a deletion.

Acceptance for this phase: a 500 response schedules the next attempt at the backoff delay, a 4xx marks dead letter immediately, and a 200 clears the queue entry.

Phase 4: Ingest and status API

The server exposes the ingest endpoint that persists the raw body bytes, and the deliveries listing endpoint that reports per-event state for operators.

// src/server.js
export function createHandler({ queue, hooks }) {
  return (req, res) => {
    if (req.method === 'POST' && /^\/hooks\/[\w-]+$/.test(req.url)) {
      const name = req.url.split('/')[2];
      if (!hooks[name]) { res.writeHead(404).end(); return; }
      const chunks = [];
      req.on('data', (c) => chunks.push(c));
      req.on('end', () => {
        const body = Buffer.concat(chunks);          // raw bytes, never re-serialized
        const id = queue.enqueue(name, body);
        res.writeHead(202, { location: `/hooks/${name}/deliveries/${id}` });
        res.end();
      });
      return;
    }
    if (req.method === 'GET' && req.url.endsWith('/deliveries')) {
      const name = req.url.split('/')[2];
      res.writeHead(200, { 'content-type': 'application/json' });
      res.end(JSON.stringify(queue.listDeliveries(name)));
    }
  };
}

The ingest contract mirrors the signer’s requirement: the body is accumulated as raw buffers and persisted as the BLOB, so what gets delivered later is byte-identical to what arrived — the 202 with a location header gives the producer a permalink to poll, and the queue position is the honest answer to “did you get it” the moment persistence happens.

The deliveries endpoint answers the two questions operators actually ask — is anything stuck, and what did the last failure say. Attempt counts, the last error string, and the next retry time come straight from the queue table, so there is no second source of truth to reconcile, and the integration test watches a delivery transition from queued to delivered through this endpoint while the receiving test server observes the signature headers.

Acceptance for this phase: posting an event returns 202 with a location header, and the deliveries endpoint transitions from queued to delivered as the dispatcher works, with signature headers visible on the receiving test server.

Design trade-offs

At-least-once over exactly-once. Exactly-once over plain HTTP requires receiver cooperation the forwarder cannot assume; stating at-least-once and signing every payload gives receivers the event identity to deduplicate, which is the contract real webhook platforms converge on. Pretending otherwise is the silent-data-loss option.

SQLite over a message broker. A broker gives horizontal scale and subscriber semantics; one file gives durable claiming in two SQL statements, zero operational dependencies, and a queue an operator can inspect with the sqlite3 shell. Throughput is bounded by design — the single dispatcher is the documented limit.

Immediate dead letter on 4xx over retry-everything. Retrying a malformed request wastes the receiver’s quota and the forwarder’s budget; classifying failures by status family is one comparison and the correct policy. The dead letter status keeps the evidence for replay.

Sweeper over transaction-spanning delivery. Holding a write transaction across an HTTP request couples database health to network health — the sweeper decouples them at the cost of a bounded re-delivery window, which is the cheaper failure.

Testing strategy

Run npx vitest. Signer tests use fixed keys and bodies with expected digests computed via the Node crypto shell, plus the one-byte mutation case. Queue tests drive claim and sweep with an injected clock and assert exactly-once claiming — the second claim returning nothing is the assertion that holds the whole concurrency story up.

Dispatcher tests run against a local stub receiver that scripts status sequences (500, then 500, then 200) and assert backoff timings, dead letter transitions, and that received bodies and signatures verify with the shared secret — the stub is the receiver, so the signature verification is real end to end. The ingest integration test posts a real event through the HTTP API and follows it to delivery, exercising the full path: raw bytes in, raw bytes out, same digest both sides.

Troubleshooting playbook

Receiver reports invalid signature. The receiver’s recomputed digest does not match the x-hook-signature header. The cause is almost always re-serialization — the forwarder parsed and re-stringified the JSON somewhere between ingest and delivery, changing key order or whitespace. The fix is the raw-buffer discipline: accumulate chunks, persist the BLOB, deliver the BLOB, sign the BLOB. The timestamp prefix must also match the header exactly, since it is part of the signed payload.

Deliveries stuck in-flight after a crash. Rows sit in in-flight forever after a process kill. The cause is a sweeper threshold longer than the operator’s patience, or the sweep never running on startup. The fix is lowering the threshold, running the sweep on startup, and ensuring the claim step always timestamps claimed_at — stale entries are only detectable if the timestamp exists.

Retries hammer a struggling receiver. Backoff fires but the receiver is still down, and the failure count climbs fast. The cause is a backoff cap set too low for the receiver’s recovery time. The fix is raising the cap for that hook — it is a config value per endpoint — and treating 429 responses with their Retry-After header as authoritative when present, since a receiver that says “wait 30 seconds” knows its own recovery curve better than any exponential guess.

Extending the project

The queue interface is the growth surface: per-hook dispatchers with ordering guarantees are a claim query scoped by hook, parallelism is N dispatcher loops over the same atomic claim, and a scheduled replay command for dead letter rows is an UPDATE back to queued plus the existing machinery. Multi-process deployments need the claim to take a database-level lock, which the single-writer SQLite constraint already provides. Subscriber fan-out — one event, many targets — is a queue write per target inside the ingest transaction.

For deployment, run as a single long-lived process or container with the SQLite file on persistent storage. Expose the status endpoint behind an internal-only listener or auth so operators can watch queue depth. Set per-hook secrets through environment files rather than hooks.json when secrets differ per environment, and alert on dead letter count rather than raw failure count — dead letters mean a hook is broken, while raw failures often just mean a receiver is redeploying. The signature envelope deliberately mirrors the conventions of the major webhook platforms so receivers can reuse their existing verification logic: GitHub prefixes the timestamp with a scheme tag and HMACs timestamp.body, Stripe splits the header into timestamp and signature fields with the same timestamp.body payload, and this project’s x-hook-timestamp plus x-hook-signature follows the same shape. That family resemblance is practical, not aesthetic — a receiver that already verifies GitHub or Stripe webhooks has the constant-time-compare habit, the replay-window check, and the raw-body middleware in place, and adapting them to one more header pair is a five-minute change. The one rule that transfers verbatim from all of those platforms: compare digests in constant time, never with === on the hex strings, because early-exit string comparison leaks timing information about the signature a patient attacker can exploit.