← Workflows

阅读中文版本

Poison Message Handling Workflow

Detect a message that repeatedly fails, cap redeliveries, route it to a dead-letter queue, and provide a safe replay path after the fix.

intermediate~60 minmessagingreliabilityoperations

Objective

Stop a single malformed message from blocking a queue while preserving it for analysis and replay.

Environment

  • A broker with redelivery and dead-letter support
  • A handler that can classify permanent failures

Steps

1. Classify failures so permanent errors do

Classify failures so permanent errors do not retry forever.

Expected state: Poison messages are identified, not retried indefinitely.

2. Set a delivery limit so

Set a delivery limit so the message moves to a dead-letter queue.

Expected state: The main queue keeps flowing.

3. Alert on dead-letter arrivals

Alert on dead-letter arrivals with the message headers.

Expected state: Operators know a message needs attention.

4. After fixing the cause, replay

After fixing the cause, replay the dead-lettered message safely.

Expected state: Data is recovered without duplication.

Decision points

Auto-retry or dead-letter immediately?

  • Dead-letter after N attempts
  • Retry forever

Dead-letter after a small bounded number; infinite retry hides bugs and blocks the queue.

Replay automatically or manually?

  • Manual after fix
  • Automatic timed replay

Manual is safer for permanent failures; automatic only for clearly transient cases.

Expected outcome

The queue is unblocked, the poison message is quarantined with context, and it is replayed once the underlying cause is fixed.

Verification

Confirm normal traffic flows while the poison message sits in the dead-letter queue, and that replay produces the expected effect exactly once.

Cleanup

Archive the dead-lettered message and record the root cause and fix.

Overview

Stop a single malformed message from blocking a queue while preserving it for analysis and replay. Confirm normal traffic flows while the poison message sits in the dead-letter queue, and that replay produces the expected effect exactly once.

Steps

  1. Classify failures so permanent errors do not retry forever. Expected state: Poison messages are identified, not retried indefinitely.
  2. Set a delivery limit so the message moves to a dead-letter queue. Expected state: The main queue keeps flowing.
  3. Alert on dead-letter arrivals with the message headers. Expected state: Operators know a message needs attention.
  4. After fixing the cause, replay the dead-lettered message safely. Expected state: Data is recovered without duplication.
channel.consume(q, (msg) => {
  try { handle(msg); channel.ack(msg); }
  catch (e) {
    if (msg.fields.redelivered && isPermanent(e)) channel.nack(msg, false, false); // -> DLQ
    else channel.nack(msg, false, true);
  }
});

Verification

  • The queue is unblocked, the poison message is quarantined with context, and it is replayed once the underlying cause is fixed.

Decision points

  • Auto-retry or dead-letter immediately? Options: Dead-letter after N attempts; Retry forever. Recommended: Dead-letter after a small bounded number; infinite retry hides bugs and blocks the queue.
  • Replay automatically or manually? Options: Manual after fix; Automatic timed replay. Recommended: Manual is safer for permanent failures; automatic only for clearly transient cases.