Dead-Letter Handling Workflow
Route failed messages to a dead-letter queue with enough context to reprocess them once the cause is fixed.
Objective
Ensure failed messages are captured, inspectable, and reprocessable instead of silently dropped.
Environment
- A message queue with dead-letter support
- Logging and alerting
- A reprocessing mechanism
Steps
1. Route failures to a DLQ
Send messages that exhaust retries to a dead-letter queue with metadata.
Expected state: No failed message is silently dropped.
2. Alert on DLQ growth
Alert when the DLQ grows beyond a threshold or rate.
Expected state: Failures become visible quickly.
3. Triage by error class
Group dead-lettered messages by error to find systemic causes.
Expected state: Common failure causes are identified.
4. Reprocess after fixing
Replay the DLQ after the cause is fixed and confirm it drains.
Expected state: Valid messages are reprocessed and the DLQ clears.
Decision points
Retry forever or dead-letter?
- Bounded retries then DLQ
- Infinite retries
Bound retries and dead-letter so poison messages cannot block the pipeline.
Auto-replay or manual?
- Manual after review
- Automatic always
Replay manually after fixing the cause; auto-replay can amplify failures.
Expected outcome
Failed messages are preserved and reprocessed, turning transient failures into recoverable events.
Verification
Confirm the DLQ captures failures with context, alerts fire on growth, and a replay drains it after a fix.
Cleanup
Retain or purge the DLQ per the data policy and record the incident.
Overview
Ensure failed messages are captured, inspectable, and reprocessable instead of silently dropped.
This workflow is reusable across teams and environments; adapt the commands to your tooling while keeping the sequence and checkpoints intact.
Environment and prerequisites
- A message queue with dead-letter support
- Logging and alerting
- A reprocessing mechanism
Procedure
1. Route failures to a DLQ
Send messages that exhaust retries to a dead-letter queue with metadata.
Expected state: No failed message is silently dropped.
2. Alert on DLQ growth
Alert when the DLQ grows beyond a threshold or rate.
Expected state: Failures become visible quickly.
3. Triage by error class
Group dead-lettered messages by error to find systemic causes.
Expected state: Common failure causes are identified.
4. Reprocess after fixing
Replay the DLQ after the cause is fixed and confirm it drains.
Expected state: Valid messages are reprocessed and the DLQ clears.
aws sqs get-queue-attributes --queue-url "$DLQ" \
--attribute-names ApproximateNumberOfMessages
Verification
Confirm the DLQ captures failures with context, alerts fire on growth, and a replay drains it after a fix.
Decision points
Retry forever or dead-letter?
- Bounded retries then DLQ
- Infinite retries
Recommendation: Bound retries and dead-letter so poison messages cannot block the pipeline.
Auto-replay or manual?
- Manual after review
- Automatic always
Recommendation: Replay manually after fixing the cause; auto-replay can amplify failures.
Common failures
- Silent drops lose data.
- Infinite retries block the queue with a poison message.
- No context makes reprocessing impossible.