Build a Dead Letter Queue Dashboard

Export dead-letter depth and message age to Prometheus, add a Grafana panel, and give operators a documented replay command.

~35 minrabbitmqobservabilitymessaging

Task

Make failed messages visible and replayable immediately instead of discovering them days later during a support escalation.

When to use this

A consumer rejects malformed orders into a dead letter queue. Nobody watches the queue because it has no dashboard, so two thousand failed orders accumulate before a customer reports a missing shipment.

Prerequisites

RabbitMQ 3.12 or newer with the management and Prometheus plugins
Prometheus and Grafana
Access to rabbitmqadmin or the management API

Steps

1. Declare a dead letter exchange, queue, and binding, then point the work queue at the DLX.

rabbitmqadmin declare exchange name=orders.dlx type=direct durable=true
rabbitmqadmin declare queue name=orders.dlq durable=true
rabbitmqadmin declare binding source=orders.dlx destination=orders.dlq routing_key=orders
rabbitmqadmin declare queue name=orders.work durable=true \
  arguments='{"x-dead-letter-exchange":"orders.dlx","x-dead-letter-routing-key":"orders"}'

Expected output: Rejected messages from orders.work are routed to orders.dlq through orders.dlx.

2. Enable the Prometheus plugin and scrape the per-queue metrics endpoint.

rabbitmq-plugins enable rabbitmq_prometheus && curl -s http://localhost:15692/metrics | grep rabbitmq_queue_messages_ready

Expected output: Metrics include rabbitmq_queue_messages_ready{queue="orders.dlq"} with the current depth.

3. Add a Grafana panel and an alert that fires when the DLQ stays non-empty.

rabbitmq_queue_messages_ready{queue="orders.dlq"} > 0

Expected output: The panel shows depth over time and the alert transitions to firing after ten minutes above zero.

4. Document and test a replay path that republishes dead letters to the work queue.

rabbitmqadmin get queue=orders.dlq count=10 ackmode=ack_requeue_false

Expected output: Ten messages are returned for inspection; a replay script republishes valid ones to orders.work with the original message id.

Troubleshooting

Problem

The DLQ metric is missing from the scrape

Solution

Confirm rabbitmq_prometheus is enabled and the scrape target points at port 15692; the management plugin is not required for metrics.

Problem

Replay creates duplicate side effects

Solution

Replay with the original message id and ensure the consumer deduplicates by id; never assign a new id during replay.

Limitations

Metrics show queue state, not the reason a message failed; the error is in consumer logs or a header.
This dashboard does not retry automatically; replay remains an operator action with idempotent consumers.

A DLQ without a dashboard is a data loss bucket

Dead letter queues are where messages go when retries are exhausted. Without visibility they become a silent graveyard. The minimum useful view is depth over time plus the age of the oldest message. Depth tells you something is failing; age tells you whether it is still failing now. Both should be on a dashboard that the on-call rotation actually opens.

Wire the metrics

The RabbitMQ Prometheus plugin exposes per-queue gauges on port 15692. Scrape it like any other target and build a panel filtered to the DLQ name. Add an alert that fires when depth is greater than zero for ten minutes, not instantly, so a single poison message does not page at 3 a.m. Include a link to the replay runbook in the alert description.

rabbitmq_queue_messages_ready{queue="orders.dlq"} > 0

Make replay safe

Replay is an operator action, not a cron job. Inspect messages first with rabbitmqadmin get, classify the failure, fix the consumer or the payload, and republish with the original message id. Consumers must deduplicate, because replay after a partial failure can deliver the same message twice. Record who replayed what and when in the incident channel.

Close the loop

Every DLQ message should have a known reason: validation failure, downstream outage, or a bug. Add a header with the failure class when rejecting, and graph depth by class. If the same class dominates for a week, the fix is in the consumer or the producer contract, not in more replay tooling.