Test Idempotency Keys End to End
Replay the same request key and assert one side effect, an identical response, and a conflict when the payload changes.
Task
Prove that retries with the same idempotency key create exactly one resource and return the original response.
When to use this
A mobile client times out and retries a payment. Both requests create a charge because the key is stored after the work, and the customer is billed twice.
Prerequisites
- Node.js 22 and a database with a unique constraint on (tenant_id, key)
- A payment or order endpoint that performs a side effect
- A test client that can send concurrent requests
Steps
1. Create the key table with a primary key that makes double claims impossible.
CREATE TABLE idempotency_keys (
tenant_id uuid NOT NULL,
key text NOT NULL,
request_hash text NOT NULL,
state text NOT NULL CHECK (state IN ('processing', 'completed')),
response_status int,
response_body jsonb,
created_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (tenant_id, key)
);
Expected output: Two concurrent inserts with the same tenant and key cannot both succeed.
2. Claim the key inside the same transaction as the side effect before doing the work.
await client.query('BEGIN');
const { rowCount } = await client.query(
`INSERT INTO idempotency_keys (tenant_id, key, request_hash, state)
VALUES ($1, $2, $3, 'processing')
ON CONFLICT (tenant_id, key) DO NOTHING`,
[tenantId, key, requestHash],
);
if (rowCount === 0) return replayStoredResponse(tenantId, key, requestHash);
Expected output: Only one request claims the key; the other reads the stored state instead of creating a charge.
3. Store the response in the same transaction and return it unchanged on replay.
const charge = await createCharge({ tenantId, amount });
await client.query(
`UPDATE idempotency_keys
SET state = 'completed', response_status = 201, response_body = $3
WHERE tenant_id = $1 AND key = $2`,
[tenantId, key, JSON.stringify(charge)],
);
await client.query('COMMIT');
Expected output: A replay with the same key and body returns 201 with the original charge id and creates no second charge.
4. Reject the same key with a different payload instead of returning the old response.
const second = await post('/api/payments', { key: 'k-1', body: { amount: 999 } });
assert.equal(second.status, 422);
assert.equal(second.body.error, 'idempotency_key_reuse');
Expected output: 422 with a stable error code, because returning the original charge for a different amount would be misleading.
5. Fire concurrent duplicates and assert exactly one side effect.
const responses = await Promise.all(
Array.from({ length: 5 }, () => post('/api/payments', { key: 'k-2', body: charge })),
);
assert.equal(await countCharges('k-2'), 1);
const ids = new Set(responses.filter((r) => r.status === 201).map((r) => r.body.id));
assert.equal(ids.size, 1);
Expected output: One charge row exists, every successful response references it, and overlapping losers receive 409 while processing.
Troubleshooting
Problem
The same key creates two resources
Solution
The key is written after the side effect, leaving a crash window. Insert the key first in the same transaction, or use a unique constraint on the effect itself.
Problem
Concurrent duplicates both proceed
Solution
Two requests read no row and both insert. Use INSERT ... ON CONFLICT DO NOTHING and check rowCount, or lock the key row with SELECT FOR UPDATE.
Problem
The replayed response differs from the original
Solution
Persist the status code, body, and important headers. Recomputing the response from current state produces a different answer after other changes.
Problem
Keys collide between endpoints
Solution
Scope the unique key by tenant and endpoint, or namespace the key value with the route.
Limitations
- Keys need a retention window; replays after expiry can create duplicates.
- The client must resend the same key; the server cannot deduplicate requests that arrive without one.
- Large or streaming responses are impractical to store; keep idempotency for command-style endpoints.
Retries are inevitable
Networks drop responses, mobile clients time out, and gateways retry. From the client’s perspective, a request that timed out is in an unknown state: it may have succeeded. Without idempotency, the safe client behaviour (retry) conflicts with the safe business behaviour (do not charge twice). An idempotency key resolves the conflict by letting the server recognize the retry and return the original outcome.
Claim before you work
The critical ordering is: insert the key, do the work, store the response, commit. If the key is written after the charge, a crash between the two leaves a charge with no key, and the retry creates a second one. Claiming first means a crash leaves a processing row; the retry sees it and returns a conflict or waits, and a cleanup job can reconcile stale rows.
INSERT INTO idempotency_keys (tenant_id, key, request_hash, state)
VALUES ($1, $2, $3, 'processing')
ON CONFLICT (tenant_id, key) DO NOTHING;
Store the exact response
Replay must return the same status and body, not a recomputed response. If the charge response included a timestamp or the current balance, recomputing it later produces a different answer and clients may treat it as a new operation. Persist status, body, and headers that affect client behaviour, and replay them byte for byte.
Detect key reuse with a different payload
The same key with a different body is a client bug or an attack. Returning the original response silently confirms the wrong amount. Compare a hash of the canonical request body and return 422 when it differs, with a clear error code. This also protects against a key collision where two different operations pick the same string.
Test concurrency, not just sequence
Sequential replay tests only prove the read path. Fire five identical requests in parallel and assert that exactly one charge exists and every successful response points to it. Concurrent duplicates are the case that actually happens during an incident, when the client retries while the first request is still in flight.