← Workflows

阅读中文版本

Dead-Letter Handling Workflow

Route failed messages to a dead-letter queue with enough context to reprocess them once the cause is fixed.

intermediate~50 mindatapipelinereliability

Objective

Ensure failed messages are captured, inspectable, and reprocessable instead of silently dropped.

Environment

  • A message queue with dead-letter support
  • Logging and alerting
  • A reprocessing mechanism

Steps

1. Route failures to a DLQ

Send messages that exhaust retries to a dead-letter queue with metadata.

Expected state: No failed message is silently dropped.

2. Alert on DLQ growth

Alert when the DLQ grows beyond a threshold or rate.

Expected state: Failures become visible quickly.

3. Triage by error class

Group dead-lettered messages by error to find systemic causes.

Expected state: Common failure causes are identified.

4. Reprocess after fixing

Replay the DLQ after the cause is fixed and confirm it drains.

Expected state: Valid messages are reprocessed and the DLQ clears.

Decision points

Retry forever or dead-letter?

  • Bounded retries then DLQ
  • Infinite retries

Bound retries and dead-letter so poison messages cannot block the pipeline.

Auto-replay or manual?

  • Manual after review
  • Automatic always

Replay manually after fixing the cause; auto-replay can amplify failures.

Expected outcome

Failed messages are preserved and reprocessed, turning transient failures into recoverable events.

Verification

Confirm the DLQ captures failures with context, alerts fire on growth, and a replay drains it after a fix.

Cleanup

Retain or purge the DLQ per the data policy and record the incident.

Overview

Ensure failed messages are captured, inspectable, and reprocessable instead of silently dropped.

This workflow is reusable across teams and environments; adapt the commands to your tooling while keeping the sequence and checkpoints intact.

Environment and prerequisites

  • A message queue with dead-letter support
  • Logging and alerting
  • A reprocessing mechanism

Procedure

1. Route failures to a DLQ

Send messages that exhaust retries to a dead-letter queue with metadata.

Expected state: No failed message is silently dropped.

2. Alert on DLQ growth

Alert when the DLQ grows beyond a threshold or rate.

Expected state: Failures become visible quickly.

3. Triage by error class

Group dead-lettered messages by error to find systemic causes.

Expected state: Common failure causes are identified.

4. Reprocess after fixing

Replay the DLQ after the cause is fixed and confirm it drains.

Expected state: Valid messages are reprocessed and the DLQ clears.

aws sqs get-queue-attributes --queue-url "$DLQ" \
  --attribute-names ApproximateNumberOfMessages

Verification

Confirm the DLQ captures failures with context, alerts fire on growth, and a replay drains it after a fix.

Decision points

Retry forever or dead-letter?

  • Bounded retries then DLQ
  • Infinite retries

Recommendation: Bound retries and dead-letter so poison messages cannot block the pipeline.

Auto-replay or manual?

  • Manual after review
  • Automatic always

Recommendation: Replay manually after fixing the cause; auto-replay can amplify failures.

Common failures

  • Silent drops lose data.
  • Infinite retries block the queue with a poison message.
  • No context makes reprocessing impossible.