Poison Message Handling Workflow
Detect a message that repeatedly fails, cap redeliveries, route it to a dead-letter queue, and provide a safe replay path after the fix.
Objective
Stop a single malformed message from blocking a queue while preserving it for analysis and replay.
Environment
- A broker with redelivery and dead-letter support
- A handler that can classify permanent failures
Steps
1. Classify failures so permanent errors do
Classify failures so permanent errors do not retry forever.
Expected state: Poison messages are identified, not retried indefinitely.
2. Set a delivery limit so
Set a delivery limit so the message moves to a dead-letter queue.
Expected state: The main queue keeps flowing.
3. Alert on dead-letter arrivals
Alert on dead-letter arrivals with the message headers.
Expected state: Operators know a message needs attention.
4. After fixing the cause, replay
After fixing the cause, replay the dead-lettered message safely.
Expected state: Data is recovered without duplication.
Decision points
Auto-retry or dead-letter immediately?
- Dead-letter after N attempts
- Retry forever
Dead-letter after a small bounded number; infinite retry hides bugs and blocks the queue.
Replay automatically or manually?
- Manual after fix
- Automatic timed replay
Manual is safer for permanent failures; automatic only for clearly transient cases.
Expected outcome
The queue is unblocked, the poison message is quarantined with context, and it is replayed once the underlying cause is fixed.
Verification
Confirm normal traffic flows while the poison message sits in the dead-letter queue, and that replay produces the expected effect exactly once.
Cleanup
Archive the dead-lettered message and record the root cause and fix.
Overview
Stop a single malformed message from blocking a queue while preserving it for analysis and replay. Confirm normal traffic flows while the poison message sits in the dead-letter queue, and that replay produces the expected effect exactly once.
Steps
- Classify failures so permanent errors do not retry forever. Expected state: Poison messages are identified, not retried indefinitely.
- Set a delivery limit so the message moves to a dead-letter queue. Expected state: The main queue keeps flowing.
- Alert on dead-letter arrivals with the message headers. Expected state: Operators know a message needs attention.
- After fixing the cause, replay the dead-lettered message safely. Expected state: Data is recovered without duplication.
channel.consume(q, (msg) => {
try { handle(msg); channel.ack(msg); }
catch (e) {
if (msg.fields.redelivered && isPermanent(e)) channel.nack(msg, false, false); // -> DLQ
else channel.nack(msg, false, true);
}
});
Verification
- The queue is unblocked, the poison message is quarantined with context, and it is replayed once the underlying cause is fixed.
Decision points
- Auto-retry or dead-letter immediately? Options: Dead-letter after N attempts; Retry forever. Recommended: Dead-letter after a small bounded number; infinite retry hides bugs and blocks the queue.
- Replay automatically or manually? Options: Manual after fix; Automatic timed replay. Recommended: Manual is safer for permanent failures; automatic only for clearly transient cases.