Add a Dead-Letter Queue Policy

Bound retries, route poison messages to a DLQ with context, alarm on depth and age, and document a safe replay procedure.

~25 minmessagingreliabilityoperations

Task

Keep a queue consumer moving when one message cannot be processed, while preserving the message for diagnosis and controlled replay.

When to use this

A malformed event is retried forever, blocks an ordered partition, and hides newer messages behind it.

Prerequisites

A queue or stream that supports redelivery or a dead-letter target
A retry limit and a classification of retryable versus permanent errors
An alerting and on-call path for the DLQ

Steps

1. Configure a bounded receive count and a dead-letter target.

{
  "RedrivePolicy": {
    "deadLetterTargetArn": "arn:aws:sqs:region:account:orders-dlq",
    "maxReceiveCount": 5
  }
}

Expected output: The consumer receives a message at most five times before it moves to the DLQ.

2. Preserve context in the DLQ message or envelope: original topic, partition, offset, error class, consumer version, and attempt count.

{
  "original": { "topic": "orders", "partition": 3, "offset": 91822 },
  "error": "schema validation failed: missing customerId",
  "consumerVersion": "1.8.2",
  "attempts": 5
}

Expected output: An operator can identify the source and failure without re-running the consumer.

3. Alarm on DLQ depth and oldest message age, and document a replay procedure that requires idempotent consumers.

aws sqs get-queue-attributes --queue-url $DLQ_URL --attribute-names ApproximateNumberOfMessages

Expected output: A growing or aging DLQ pages the owner before the main queue is affected.

Troubleshooting

Problem

Transient errors fill the DLQ

Solution

Retry connection errors and 5xx responses with backoff; route only exhausted or permanent failures to the DLQ.

Problem

Replay duplicates state

Solution

Make the consumer idempotent and replay into a staging consumer before production.

Limitations

A DLQ does not repair the cause; the producer or schema still needs a fix.
Replay may lose ordering; check whether the consumer depends on sequence.

Bound the retries

A poison message should not block a queue forever. Set a maximum receive count and a dead-letter target, and classify failures before retrying. A timeout or a 503 may succeed later; a schema mismatch or an authorization failure will not. Route permanent failures immediately and transient failures through a delayed retry.

Make the DLQ observable

A DLQ that nobody watches is a silent data-loss queue. Alarm on message count and oldest age, and include enough context to diagnose the failure. Preserve the original payload, the error class, the consumer version, and the source position. Redact secrets before storing context.

{
  "original": { "topic": "orders", "partition": 3, "offset": 91822 },
  "error": "schema validation failed: missing customerId",
  "consumerVersion": "1.8.2",
  "attempts": 5
}

Verify the policy

Expected outcome: a poison message reaches the DLQ after exactly five receives, the main queue continues processing newer messages, and the DLQ alarm fires on depth or age. Confirm by publishing a malformed message followed by a valid one and checking that the valid message is processed while the malformed one is parked with its source context. Replay the DLQ only after the consumer is idempotent and the cause is fixed.

Replay is a release

Replay only after the cause is fixed. Make consumers idempotent, replay into staging first, and record who replayed what. Check ordering requirements and downstream rate limits. A replay that doubles a counter or resends an email is a new incident, not a recovery.