Add Idempotency Keys to a Write Endpoint

Accept an Idempotency-Key header, store the request fingerprint and response, and replay the original result for retries instead of creating duplicates.

~30 minapi-designreliabilityhttp

Task

Make a non-idempotent POST endpoint safe to retry by implementing idempotency-key handling.

When to use this

A client retries a payment or order request after a timeout. Without a key, the server cannot tell whether the first attempt committed.

Prerequisites

A database or cache with atomic unique constraints
A write endpoint that may be retried by clients
A retention policy for stored keys

Steps

1. Require a client-generated idempotency key with a bounded length and reject missing or malformed values for write endpoints.

const key = req.get('Idempotency-Key');
if (!key || key.length < 16 || key.length > 200) {
  return res.status(400).json({ error: 'Idempotency-Key required' });
}

Expected output: Requests without a valid key are rejected before any side effect.

2. Store the key, a hash of the canonical request body, and the response under a unique constraint. Insert before executing the side effect.

INSERT INTO idempotency_keys (key, request_hash, status)
VALUES ($1, $2, 'in_progress')
ON CONFLICT (key) DO NOTHING
RETURNING key;

Expected output: The first request claims the key; a concurrent duplicate sees no returned row.

3. If the key exists with the same request hash, return the stored response. If the hash differs, return 409 and do not execute the side effect.

Expected output: Retries receive the original result; conflicting payloads are rejected.

4. Store the final status and response, and expire keys after the client's retry window. Verify by sending the same request twice.

curl -X POST -H 'Idempotency-Key: 01J...' -d @order.json https://api.example.com/orders

Expected output: The second request returns the same order ID and does not create a second order.

Troubleshooting

Problem

Two concurrent requests both execute

Solution

Rely on the unique constraint and an in-progress state; use a transaction or lock so only one worker proceeds.

Problem

Keys grow without bound

Solution

Add an expiry column and a scheduled cleanup job; keep the window longer than the client's maximum retry period.

Limitations

The key identifies a retry, not a user; never use it as authentication.
Storing full responses can leak sensitive data; store only what is needed to replay safely.

The retry contract

A client sends Idempotency-Key: <opaque value> with a write request. The server treats the key as the identity of that logical operation. Repeating the request with the same key and the same payload must not create a second effect.

CREATE TABLE idempotency_keys (
  key text PRIMARY KEY,
  request_hash text NOT NULL,
  status text NOT NULL,
  response jsonb,
  created_at timestamptz NOT NULL DEFAULT now(),
  expires_at timestamptz NOT NULL
);

Use INSERT ... ON CONFLICT DO NOTHING RETURNING to claim the key atomically. If the insert returns nothing, load the existing record and compare request_hash.

Verify

The first request creates the resource and stores a response.
Repeating it with the same key returns the same resource ID.
Reusing the key with a different body returns 409.
Concurrent duplicates produce exactly one side effect.
Expired keys are cleaned up and no longer replay.

Scope

Apply idempotency only to non-idempotent writes such as orders, payments, and webhook-triggered actions. GET, PUT, and DELETE are already idempotent by HTTP semantics. Keep the key opaque, bound its length, and retain it longer than the client’s retry window.