Queue Consumer Lag Incident Workflow
Treat consumer lag as an incident with its own triage: separate producer surge from consumer stall, then scale, shed, or drain accordingly.
Objective
Resolve consumer lag incidents by identifying whether the producer or consumer side is the cause, protecting downstream dependencies while draining the backlog.
Environment
- A message queue with per-consumer lag metrics
- Consumer autoscaling or manual scaling capability
- Visibility into producer rates and downstream dependency health
Steps
1. Read the lag shape
Distinguish the lag curve: a step function (consumer stopped) from a ramp (producer surge) from sawtooth (consumer slower than sustained rate).
Expected state: The lag pattern is classified, which names the half of the system to investigate.
2. Check consumer health
Inspect consumer errors, restarts, and processing time per message for a step pattern; a crashed or poison-message-looping consumer is the cause.
Expected state: For step patterns, the specific consumer failure is identified before any scaling decision.
3. Check producer and downstream for ramps
For ramp patterns, verify the producer surge is legitimate (traffic) versus buggy (retry storm), and confirm downstream dependencies can absorb faster consumption.
Expected state: Scaling decisions account for downstream capacity, not just queue depth.
4. Drain with protection
Scale consumers to drain while rate-limiting against downstream limits; skip or dead-letter non-critical topics to prioritize the backlog that matters.
Expected state: Lag decreases monotonically while downstream latency stays within bounds.
5. Fix the root and recalibrate alerts
Fix the consumer bug or the producer storm, then review the lag alert threshold against legitimate surge patterns to prevent repeat false pages.
Expected state: The root cause is fixed, and the alert policy distinguishes surge from failure going forward.
Expected outcome
Lag incidents resolved with the correct side fixed, backlogs drained without crushing downstream services, and alert thresholds that stop paging on legitimate surges.
Verification
Lag returns to near zero and stays there through the next traffic peak; the alert history shows no repeat false-positive pages for the same surge pattern.
Lag is a symptom with two possible parents
Consumer lag gets treated as one problem, but a stopped consumer and a producer surge are opposites: the first needs a fix, the second needs capacity or shedding. Reading the lag curve first directs the response correctly.
lag shapes (5-min buckets):
step: 0,0,0,41200,41300,41300 -> consumer stopped
ramp: 0,900,3100,8400,17000 -> producer surge
sawtooth: 0,8K,2K,9K,3K,10K -> consumer < rate
step: consumer restart x 14, poison msg loop
fix: dead-letter the poison message, patch handler
ramp: marketing email blast (legit)
fix: scale consumers 4->12, cap at db limit (200/s)
skip analytics topic (non-critical)Draining has a blast radius
Scaling consumers to drain a backlog fires the entire backlog at downstream services in a compressed window. A database that survived 200 writes per second will not survive 5,000 arriving at once. Rate caps and topic prioritization are part of the drain, not optional niceties.
Recalibrate after every incident
If the lag alert fired for a legitimate surge and the response was scale and wait, the threshold is wrong. Each lag incident should end with an explicit decision: what level of surge is normal business, and what level means broken — encoded back into the alert.