阅读中文版本

Build an Outbound Webhook Delivery Service

A complete outbound webhook delivery service: signed payloads, per-endpoint retries with backoff, circuit breaking, and delivery log tests.

webhook-delivery/
├── src/worker.ts
├── src/sign.ts
├── src/retry.ts
├── src/breaker.ts
├── tests/delivery.test.ts
└── package.json

const sig = hmac(secret, ts + '.' + body);
headers['x-signature'] = 't=' + ts + ',v1=' + sig;

dueAt = now + Math.min(maxDelay, base * 2 ** attempt);

const ac = new AbortController(); setTimeout(() => ac.abort(), timeoutMs);

if (breaker.isOpen(endpointId)) return defer(delivery, cooldownMs);

await db.insertAttempt({ deliveryId, status, code, ms });

Project overview

Delivering a webhook means making an HTTP request to someone else’s server, and that server will be slow, down, or hostile at some point. This build signs the payload, retries transient failures with backoff, and stops hammering an endpoint that keeps failing.

You will persist deliveries, sign the payload so the receiver can verify it, POST with a timeout, classify the response, reschedule transient failures with backoff, and open a circuit breaker per endpoint after repeated failures.

The circuit breaker is what keeps a broken subscriber from consuming delivery capacity indefinitely. Without it, one dead endpoint quietly consumes the worker pool that every other subscriber depends on.

Signing is what lets the receiver trust the payload, and including a timestamp in the signature is what stops a captured delivery from being replayed later, because the receiver can reject a stale timestamp.

Each phase adds one property: signing, retry, the timeout, the breaker, and the delivery log, so delivery is reliable and bounded at every stage.

It is for teams that send events to customer endpoints and have learned that one broken receiver can degrade delivery for everyone. The service bounds each delivery in time and isolates a failing endpoint with its own breaker.

It is also a compact lesson in treating another service as untrusted by default: cap its time, cap its response, classify its failures, and never let one subscriber’s problem become every subscriber’s problem.

What you will build

The service provides five capabilities, each with a test that uses a stub receiver:

• Sign the payload
• Retry with backoff
• Delivery timeout
• Circuit breaker
• Delivery log

A signed payload lets the receiver verify authenticity; retry with backoff handles transient failure; a delivery timeout bounds a slow endpoint; a circuit breaker defers deliveries to a failing endpoint; and a delivery log records every attempt for the subscriber to inspect.

The design assumes the receiver is unreliable and possibly hostile, so each delivery is bounded in time, classified explicitly, and attributed to a subscriber that can see its own status.

Architecture and data model

A delivery worker pulls due deliveries, signs the payload, and POSTs it with a timeout. Responses are classified; transient failures reschedule with backoff, and a per-endpoint circuit breaker opens after repeated failures.

A delivery worker pulls due deliveries, signs the payload with a per-subscriber secret and a timestamp, POSTs it with a timeout, and classifies the response.

Transient failures reschedule with exponential backoff and jitter, while permanent failures stop, so a receiver that rejects a payload does not receive it forever.

A per-endpoint circuit breaker opens after a run of failures and defers further deliveries until a probe succeeds, which contains the blast radius of one broken subscriber.

Every attempt is recorded with the status and the response summary, so a subscriber can self-diagnose rather than opening a support ticket for each failure.

The per-subscriber secret is the unit of trust, so a subscriber can rotate its secret without affecting any other receiver; that isolation is what keeps a single compromised receiver from undermining every other integration.

The delivery log is deliberately per subscriber and bounded, so a receiver can see its own history without exposing delivery data for other tenants.

Prerequisites and dependencies

You need storage for deliveries and per-subscriber secrets, a worker pool, and a way for subscribers to see their own delivery status.

• Node.js 20 LTS or later
• PostgreSQL 15+
• One or more test subscriber endpoints

The dependencies cover the HTTP surface, the signing, the store, and the outbound POST:

• undici (^6.19.0) — HTTP delivery with timeouts
• pg (^8.12.0) — Delivery queue and logs

pnpm init
pnpm add undici pg
pnpm add -D typescript tsx vitest

Generate a per-subscriber signing secret and store it in a secret manager, because a shared secret across subscribers means one leak compromises every receiver. Record the retry policy and the breaker threshold in configuration.

Directory structure

Separate signing, the retry policy, the delivery timeout, the circuit breaker, and the delivery log so each is testable with a stub receiver:

webhook-delivery/
├── src/worker.ts
├── src/sign.ts
├── src/retry.ts
├── src/breaker.ts
├── tests/delivery.test.ts
└── package.json

Phase 1: Sign the payload

HMAC the timestamp and body and send the signature header.

const sig = hmac(secret, ts + '.' + body);
headers['x-signature'] = 't=' + ts + ',v1=' + sig;

Subscribers can verify authenticity and reject tampering.

Sign the payload with a per-subscriber secret and a timestamp, and include both in a header, so the receiver can verify authenticity and reject a replay.

Phase 2: Retry with backoff

Reschedule transient failures with exponential backoff and a cap.

dueAt = now + Math.min(maxDelay, base * 2 ** attempt);

Transient receiver outages should not lose events.

Retry a transient failure with exponential backoff and jitter, and cap the number of attempts, because an immediate retry turns a brief outage at the receiver into a retry storm.

Phase 3: Delivery timeout

Abort a delivery that exceeds the deadline.

const ac = new AbortController(); setTimeout(() => ac.abort(), timeoutMs);

A hanging endpoint must not hold worker slots.

Apply a delivery timeout that is shorter than the worker’s patience, so a slow endpoint is classified as a transient failure rather than holding a worker indefinitely.

Phase 4: Circuit breaker

Open after N consecutive failures and probe after a cooldown.

if (breaker.isOpen(endpointId)) return defer(delivery, cooldownMs);

A dead endpoint should be skipped instead of retried forever.

Open a per-endpoint circuit breaker after a run of failures and defer deliveries until a probe succeeds, so one broken subscriber does not consume the shared pool.

Phase 5: Delivery log

Store each attempt with status, latency, and response code.

await db.insertAttempt({ deliveryId, status, code, ms });

Subscribers need to debug missed events with evidence.

Record every attempt with the status and a bounded response summary, and expose it per subscriber, so integrators can see and fix their own failures.

Testing

Point deliveries at a stub that fails twice then succeeds and assert eventual success; open the circuit with repeated failures and assert deliveries are deferred, not attempted.

Point a delivery at a stub that fails twice and then succeeds, and assert eventual success, which verifies the retry and backoff path end to end.

Open the circuit with repeated failures and assert that further deliveries are deferred rather than attempted, because a breaker that keeps trying is not a breaker.

Assert that a replay of an old signed payload is rejected by a receiver that checks the timestamp, which pins the anti-replay property.

Assert that a permanent failure is not retried, since retrying a rejected payload only delays the subscriber seeing the real problem.

Drive the retry test with a stub that records the attempt timestamps, so the backoff is verified as an increasing interval rather than merely as eventual success.

Assert that the delivery log records a response summary rather than the full response body, because storing a receiver’s error page is both a privacy and a storage problem.

pnpm vitest run

Deployment and operations

Run workers as replicas and keep concurrency modest so a slow endpoint cannot consume all slots. Expose subscriber-level delivery stats so integrators can self-diagnose.

Run workers as replicas and keep concurrency modest, so a slow endpoint cannot consume all the delivery slots.

Expose subscriber-level delivery statistics, because self-service diagnostics reduce support load and surface problems before they become tickets.

Monitor the open-breaker count and the retry rate, since a rising open count means several subscribers are down and the retry load is growing.

Cap the total outbound rate, because a burst of retries to many endpoints is indistinguishable from an attack and can trip your own egress limits.

Keep the outbound rate capped and the concurrency modest, because a retry storm generated by your own service can look like an attack to the receivers and trip their protections.

Expose a manual way to close an open breaker after a fix, because a breaker that only reopens on a probe can leave a recovered subscriber waiting longer than necessary.

Security considerations

Sign every payload, enforce HTTPS endpoints, and block private network targets to prevent server-side request forgery. Rotate signing secrets with an overlap window.

Sign with a per-subscriber secret and never log the secret or the full signed payload, since a leaked secret lets anyone forge a delivery to that subscriber.

Include a timestamp in the signature so the receiver can reject a replay, because a valid signature on an old payload is otherwise indistinguishable from a new one.

Validate the subscriber endpoint at registration, rejecting private and loopback addresses, because a webhook endpoint is a URL you will call on someone’s behalf.

Bound the response body you read from the receiver, so a hostile endpoint cannot exhaust memory by returning an enormous response.

Reject a subscriber endpoint in a private range at registration time and re-check before delivery, because an internal URL registered as a webhook is a way to probe your network from the outside.

Bind the signature algorithm and header names in the subscriber registration, so a receiver knows exactly what to verify and a change is an explicit event.

Enforce the outbound timeout on the connection and the response read together, because a timeout that covers only the request still lets a receiver stream a response forever.

Document the signature header names for receivers and require them to verify before acting, because a signature that nobody checks provides no protection at all.

Observability

Track success rate per endpoint, retry volume, circuit state, and delivery latency. Alert when an endpoint stays open, which means a broken integration.

Track delivery success rate, retry rate, and open breakers per subscriber, because the three together describe both the health of the system and of each integration.

Chart delivery latency, since a rising p99 usually means one endpoint is slow and holds slots from the others.

Measure the dead-letter rate, because a growing count is work that will never be delivered without operator intervention.

Alert on a rising open-breaker count, as it usually indicates a correlated outage rather than independent subscriber problems.

Track the age of the oldest undelivered event, because a growing age means the retry window is being exhausted for some subscriber, which is the point at which delivery becomes a support problem.

Measure the fraction of deliveries that open a breaker, because a small number of flaky endpoints can dominate the retry load far beyond their share of subscribers.

Track the average number of attempts per delivery, because a rising average means receivers are becoming less reliable overall rather than one subscriber failing.

Track the per-subscriber retry share too, because a subscriber whose retries dominate the pool is a capacity problem even while its breaker remains closed.

Troubleshooting

Subscribers receive duplicates. They do not deduplicate by delivery id; at-least-once delivery requires it.

Worker slots all blocked. A slow endpoint consumed every slot; add a timeout and cap per-endpoint concurrency.

If deliveries to one endpoint always fail, check the signature verification on the receiver, because a receiver that expects a different header or algorithm rejects every delivery.

If retries never stop, check the classification, because treating a permanent rejection as transient produces exactly that loop.

If a subscriber reports missing events, check the breaker state first, because a delivery deferred by an open breaker looks exactly like a delivery that was never sent.

If the breaker never closes, check the probe implementation, because a probe that never runs or always fails leaves the endpoint permanently deferred.

Limitations and next steps

• Delivery is at-least-once; subscribers must deduplicate.
• No guaranteed ordering.
• One signing scheme, no per-subscriber algorithms.

Delivery is push-only and does not offer a pull API, so a subscriber that was offline for a long period relies on the retry window rather than on a backlog it can read.

The circuit breaker is per endpoint and does not coordinate across subscribers, so several independent failures still consume shared capacity until each breaker opens.

There is no payload transformation per subscriber, so every subscriber receives the same event shape and adapts on its side.

The service does not verify that a subscriber’s endpoint remains authorized after registration, which a periodic recheck would add.

There is no subscriber-facing replay tool, so re-sending a range of events requires an operator action rather than a self-service feature.

There is no payload schema versioning beyond a field in the body, so a receiver must tolerate additive changes rather than relying on a formal contract.