Add an Outbox Relay

Publish events reliably by writing them to an outbox in the business transaction, then relaying unpublished rows with retries and idempotent consumers.

~40 minmessagingdata-consistencyreliability

Task

Avoid the dual-write problem where a database commit succeeds but the event publish fails, or the publish succeeds but the transaction rolls back.

When to use this

An order service commits an order and then calls a broker. A crash between the two steps loses the event, and a broker outage leaves the database and downstream systems permanently inconsistent.

Prerequisites

A transactional database for business data
An outbox table in the same database
A broker or event bus and idempotent consumers

Steps

1. Write the business row and the outbox event in one transaction.

BEGIN;
INSERT INTO orders (id, customer_id, total) VALUES ($1, $2, $3);
INSERT INTO outbox (aggregate_id, event_type, payload)
VALUES ($1, 'OrderCreated', $4);
COMMIT;

Expected output: Either both rows exist or neither does.

2. Run a relay that claims unpublished rows, publishes them, and marks them published with a bounded retry.

const rows = await db.query(
  `SELECT id, payload FROM outbox
    WHERE published_at IS NULL
    ORDER BY id
    LIMIT 100
    FOR UPDATE SKIP LOCKED`
);
for (const row of rows.rows) {
  await broker.publish('orders', row.payload);
  await db.query('UPDATE outbox SET published_at = now() WHERE id = $1', [row.id]);
}

Expected output: Unpublished rows are delivered; a publish failure leaves the row unpublished for the next attempt.

3. Test a broker outage and a duplicate publish, and verify consumers are idempotent.

node --test outbox.test.mjs

Expected output: Events are eventually delivered and applying the same event twice has one effect.

Troubleshooting

Problem

Events are published out of order

Solution

Relay rows in id order per aggregate and process one aggregate at a time, or include a sequence and let consumers reorder.

Problem

The outbox grows without bound

Solution

Archive or delete published rows after a retention window and monitor the oldest unpublished age.

Limitations

Delivery is at-least-once; consumers must deduplicate by event id.
Polling adds latency and load; change data capture is an alternative for high volume.

The dual-write problem

A service writes to its database and then publishes an event. If the process crashes between the two, the event is lost. If the publish happens first and the transaction rolls back, consumers see an event for a row that does not exist. No amount of retrying the publish or the commit alone fixes this because the two systems do not share a transaction.

CREATE TABLE outbox (
  id bigserial PRIMARY KEY,
  aggregate_id text NOT NULL,
  event_type text NOT NULL,
  payload jsonb NOT NULL,
  created_at timestamptz DEFAULT now(),
  published_at timestamptz
);

Relay with claim and retry

The relay reads unpublished rows with FOR UPDATE SKIP LOCKED, publishes, and marks each row. Multiple relay workers can run without claiming the same row. A publish failure leaves the row unpublished for a later attempt. Monitor the oldest unpublished timestamp; a growing backlog means the broker or relay is failing.

Consumers must deduplicate

The relay can publish a row after a crash before marking it, so delivery is at-least-once. Consumers must record event ids or use naturally idempotent operations. The outbox gives reliable delivery, not exactly-once effects.

Verify the failure paths

Stop the broker, insert an order, and confirm the event is delivered after the broker returns. Publish the same event twice and confirm one consumer effect. Check that the outbox row and the business row commit or roll back together.