Add an Idempotent Webhook Handler
Verify webhook signatures over the raw body, deduplicate event ids in a transaction, and return quickly so provider retries cannot apply the same event twice.
Task
Receive provider webhooks safely when the provider retries, delivers out of order, or sends the same event more than once.
When to use this
A payment provider retries a webhook after a timeout. The first request already changed the order state, and the retry would charge or fulfil it twice.
Prerequisites
- A webhook signing secret shared with the provider
- The raw request body available before JSON parsing
- A database table with a unique constraint on the provider event id
Steps
1. Capture the raw body, verify the signature with a constant-time comparison, and reject stale timestamps.
app.post('/webhooks/payments',
express.raw({ type: 'application/json' }),
async (req, res) => {
const signature = req.header('x-signature') ?? '';
const expected = crypto
.createHmac('sha256', process.env.WEBHOOK_SECRET)
.update(req.body)
.digest('hex');
const valid = signature.length === expected.length &&
crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected));
if (!valid) return res.status(401).end();
const event = JSON.parse(req.body.toString('utf8'));
// continue with the event
});
Expected output: A request with an invalid signature returns 401 and performs no state change.
2. Insert the event id with a unique constraint in the same transaction as the business update, and treat a duplicate as already processed.
await db.query('BEGIN');
try {
await db.query(
'INSERT INTO processed_events (event_id, provider) VALUES ($1, $2)',
[event.id, 'payments']
);
await applyPaymentEvent(event, db);
await db.query('COMMIT');
} catch (error) {
await db.query('ROLLBACK');
if (error.code === '23505') return res.status(200).json({ received: true });
throw error;
}
Expected output: The first delivery applies the event; a retry with the same id returns 200 without applying it again.
3. Return a 2xx quickly and move slow work to a queue or background job so the provider does not time out and retry unnecessarily.
await queue.publish('payment-events', event);
res.status(202).json({ accepted: true });
Expected output: The provider receives a fast acknowledgement while processing continues asynchronously.
Troubleshooting
Problem
Signature verification fails for valid events
Solution
The body was parsed and re-serialized before hashing. Verify against the exact raw bytes the provider signed.
Problem
Duplicate events still apply twice
Solution
The dedup insert is outside the business transaction or the unique constraint is missing. Check both the table definition and the transaction boundary.
Limitations
- A unique event id protects against duplicate delivery, not out-of-order events. Use the provider's event sequence or resource version when ordering matters.
- A 2xx acknowledges receipt; if queuing fails after the acknowledgement, the event can be lost. Insert the queue record transactionally or use an outbox.
Retries are normal
Providers retry webhooks because networks fail and processes restart. A handler that assumes exactly-once delivery will eventually double-charge, double-send, or double-create. The provider’s event id is the idempotency key. Store it with a unique constraint and make the business update and the dedup insert commit together.
CREATE TABLE processed_events (
provider text NOT NULL,
event_id text NOT NULL,
processed_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (provider, event_id)
);If the insert fails with a unique violation, the event was already processed. Return success, not an error; the provider should stop retrying. If the business update fails, roll back both the update and the insert so a later retry can try again.
Verify before parsing
Signature verification must use the raw body. A framework that parses JSON and re-serializes it changes whitespace and key order, so the HMAC no longer matches. Mount the raw-body middleware only on the webhook route, verify the signature, then parse. Compare in constant time and reject timestamps outside a small window to limit replay.
Acknowledge fast, process safely
Providers time out quickly. Do the minimum durable work in the request, enqueue the rest, and return 2xx. If the queue publish can fail after the acknowledgement, use an outbox row written in the same transaction as the dedup record. That way the event is either fully accepted or can be retried without duplicating the business effect.
Test the retry
Send the same signed payload twice and assert one state change and two 2xx responses. Send a payload with a bad signature and assert 401 with no row. Send events out of order if the provider does not guarantee ordering, and verify the business logic rejects or buffers stale updates.