Kafka Consumer Lag — Find Stuck Partitions and Rebalance Storms

Diagnose growing Kafka consumer group lag with consumer-group tooling, partition counts, poll interval limits, and processing time measurements before adding consumers.

advanceddebuggingkafkastreamingperformance

Symptoms

Consumer group lag grows steadily and never returns to zero
Some partitions have consumers while others report no consumer id
The consumer group rebalances repeatedly and throughput collapses during each rebalance
Consumer logs show CommitFailedException or member removed from group

Possible causes

There are fewer active consumers than partitions, so some partitions are idle
Per-record processing is slower than the arrival rate
A single poll batch takes longer than max.poll.interval.ms, causing the member to be evicted
A poison message fails repeatedly and blocks its partition
A downstream dependency applies backpressure to the consumer

Diagnosis steps

Describe the consumer group and read lag per partition

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group orders-worker

Interpretation

LAG per partition shows hot partitions. An empty CONSUMER-ID for a partition means no member is assigned, usually because consumers are fewer than partitions.

Compare partition count with active consumers

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders | head -5

Interpretation

PartitionCount is the maximum useful consumer parallelism for a plain consumer group. Extra consumers stay idle.

Look for rebalance and poll timeout evidence

grep -E 'Rebalance|max.poll.interval|CommitFailedException|Attempt to heartbeat' logs/consumer.log | tail -40

Interpretation

Repeated rebalances shortly after start point to poll interval overruns or session timeout, not raw throughput.

Measure processing time per batch

grep -E 'poll returned [0-9]+ records|processing batch' logs/consumer.log | tail -20

Interpretation

Compare the elapsed time between poll batches with max.poll.interval.ms. A batch that takes longer than the limit will trigger eviction even if lag is low.

Root cause

The consumer group cannot apply enough parallel processing to the incoming record rate, or members leave the group before committing. Lag is the symptom: either partitions have no consumer, processing per record exceeds the arrival rate, or poll interval overruns trigger rebalances that repeatedly restart progress.

Fix

Raise partition count and consumers together up to the partition count, reduce per-record work or batch size, and keep a single poll loop shorter than max.poll.interval.ms. Handle poison messages with a dead-letter topic, use cooperative-sticky assignment, and monitor lag per partition instead of only the group total.

Risk note

Increasing partitions breaks ordering guarantees for existing keys because the key-to-partition mapping changes. Enlarging max.poll.interval.ms delays detection of genuinely dead consumers. Auto-commit can skip records that were fetched but not processed.

Verify the fix

Watch the group description over a full traffic cycle: lag falls toward zero, no partition stays unassigned, and the rebalance count stops increasing. Verify P99 batch processing stays below max.poll.interval.ms and that committed offsets advance steadily.

Lag is a queue, not an error rate

A consumer group’s lag is the distance between the latest produced offset and the last committed offset. It can grow for three different reasons, and each has a different fix: missing parallelism, slow processing, or members repeatedly leaving the group.

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group orders-worker

Read the output column by column. LAG per partition identifies hot partitions; a blank CONSUMER-ID means the partition has no owner. CURRENT-OFFSET advancing slowly while lag is stable indicates throughput is matched but behind, whereas a frozen offset points at a stuck partition or poison message.

Check parallelism before scaling

PartitionCount from kafka-topics.sh --describe is the upper bound on useful consumers in a classic group. Adding a fourth consumer to a three-partition topic only creates an idle member. Scaling beyond that requires more partitions, which changes key ordering for existing keys.

Rule out poll interval overruns

The consumer is evicted when a single poll() loop takes longer than max.poll.interval.ms. Check the consumer log for Rebalance or CommitFailedException shortly after startup, and measure how long a batch takes. Reducing max.poll.records and processing in bounded chunks is often more effective than adding consumers.

max.poll.records=200
max.poll.interval.ms=300000
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

A failed record should be routed to a dead-letter topic with its original topic, partition, and offset, then committed so the partition can move forward.

Verify recovery

Track per-partition lag and rebalance count together for at least one full traffic cycle. Lag should trend to zero, every partition should show a consumer id, and processing time per batch should stay comfortably below the poll interval. If lag is flat while offsets advance, the group is keeping up; if lag is flat while offsets do not move, the partition is blocked.