Alert on Queue Backlog Age, Not Depth

Page on ApproximateAgeOfOldestMessage with a CloudWatch alarm so a stalled consumer is detected before the queue depth explodes.

~25 minawsqueuesalerting

Task

Detect a stuck or too-slow consumer within minutes using message age rather than an arbitrary depth threshold.

When to use this

An SQS consumer stops processing after a bad deploy. The queue has a depth alarm at 10,000 messages, but traffic is only 200 messages per minute, so the alarm stays green for almost an hour while orders are delayed.

Prerequisites

An SQS queue with a consumer service
CloudWatch and SNS configured for paging
AWS CLI v2 or Terraform for alarm management

Steps

1. Create an alarm on the age of the oldest message, using five one-minute periods to avoid flapping.

aws cloudwatch put-metric-alarm \
  --alarm-name orders-queue-oldest-age \
  --namespace AWS/SQS \
  --metric-name ApproximateAgeOfOldestMessage \
  --dimensions Name=QueueName,Value=orders-processing \
  --statistic Maximum \
  --period 60 \
  --evaluation-periods 5 \
  --threshold 300 \
  --comparison-operator GreaterThanThreshold \
  --treat-missing-data breaching \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:oncall-pager

Expected output: The alarm enters ALARM when the oldest unprocessed message stays older than 300 seconds for five minutes.

2. Keep a depth alarm as a secondary guard for sudden traffic spikes.

aws cloudwatch put-metric-alarm --alarm-name orders-queue-depth --namespace AWS/SQS --metric-name ApproximateNumberOfMessagesVisible --dimensions Name=QueueName,Value=orders-processing --statistic Maximum --period 60 --evaluation-periods 3 --threshold 10000 --comparison-operator GreaterThanThreshold --alarm-actions arn:aws:sns:us-east-1:123456789012:oncall-pager

Expected output: A second alarm covers bursts that raise depth quickly, while the age alarm covers slow drains.

3. Test the age alarm by pausing the consumer and sending a message.

aws sqs send-message --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/orders-processing --message-body '{"orderId":"test-1"}'

Expected output: With the consumer paused, the alarm moves to ALARM after roughly five minutes and the SNS topic pages on-call.

4. Add the runbook URL and consumer name to the alarm description for the responder.

aws cloudwatch put-metric-alarm --alarm-name orders-queue-oldest-age --alarm-description 'Consumer orders-worker stalled; runbook https://runbooks.example.com/orders-backlog' --namespace AWS/SQS --metric-name ApproximateAgeOfOldestMessage --dimensions Name=QueueName,Value=orders-processing --statistic Maximum --period 60 --evaluation-periods 5 --threshold 300 --comparison-operator GreaterThanThreshold --alarm-actions arn:aws:sns:us-east-1:123456789012:oncall-pager

Expected output: The alarm page links directly to the backlog runbook and names the owning consumer.

Troubleshooting

Problem

The alarm stays OK when the consumer is down

Solution

Set TreatMissingData to breaching for active queues; without it, a metric gap is treated as healthy.

Problem

The alarm flaps during normal peaks

Solution

Raise the threshold above the p99 processing delay or require more evaluation periods; keep the depth alarm for genuine spikes.

Limitations

ApproximateAgeOfOldestMessage is approximate and updates at coarse granularity.
The alarm identifies a stalled queue, not which consumer instance is responsible; check consumer metrics next.

Depth is a lagging signal

Queue depth depends on both arrival rate and processing rate. At low traffic, a completely stopped consumer may take an hour to cross a fixed depth threshold. Message age measures the actual user impact: how long the oldest item has been waiting. A 300-second age threshold catches a stall in five minutes regardless of traffic volume.

Configure the pair

Use the age alarm as the primary page and keep a depth alarm for sudden spikes. Set --treat-missing-data breaching so a broken metric pipeline does not silently disable the alarm. Require multiple evaluation periods to avoid paging on a single slow poll, but keep the total detection window within your delay budget.

aws cloudwatch put-metric-alarm \
  --alarm-name orders-queue-oldest-age \
  --namespace AWS/SQS \
  --metric-name ApproximateAgeOfOldestMessage \
  --dimensions Name=QueueName,Value=orders-processing \
  --statistic Maximum --period 60 --evaluation-periods 5 \
  --threshold 300 --comparison-operator GreaterThanThreshold \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:oncall-pager

Test the path

Pause the consumer, send one message, and watch the alarm transition. This validates the metric, the threshold, the SNS subscription, and the pager integration in one exercise. Then resume the consumer and confirm the alarm returns to OK. Repeat after changing queue names or consumer groups, because a renamed queue silently detaches from the alarm.

Act on the page

The first responder should check consumer health, recent deploys, and error rates, not just queue depth. A growing age with a healthy consumer usually means throughput is below arrival rate and needs scaling. Flat age with growing depth means the consumer is processing but producing more work than it completes; inspect retries and visibility timeouts.