Kafka rebalance storm — Why Consumer Groups Keep Rebalancing

Diagnose Kafka rebalance storms by reading coordinator logs and consumer group generation changes, then tune poll intervals, batch sizes, and static membership.

advanceddebuggingkafkaconsumer-groups

Symptoms

Consumer group logs show repeated Rebalance began and Revoking previously assigned partitions
Throughput stops for seconds every few minutes and then resumes
The generation number in consumer logs keeps increasing
Messages that were in flight are processed twice after each rebalance

Possible causes

A consumer spends longer than max.poll.interval.ms processing a batch between poll calls
max.poll.records is too high for the per-record processing time
Consumers join and leave because of crashes, deploys, or autoscaling churn
session.timeout.ms is shorter than the time the consumer needs to answer heartbeats

Diagnosis steps

Watch the group generation and member assignments

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group orders

Interpretation

Run it twice a minute apart. A changing generation or moving partition owners without a deploy confirms a rebalance loop. Lag columns show whether consumption actually makes progress.

Check the group state and members

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group orders --state --members --verbose

Interpretation

A group stuck in PreparingRebalance or CompletingRebalance has members that keep disappearing. Verbose members show client IDs, hosts, and assignment sizes.

Find the reason in the consumer log

kubectl logs deploy/orders-consumer --since=30m | grep -E 'Rebalance|max.poll.interval|poll timeout|LeaveGroup|Heartbeat' | tail -50

Interpretation

The message 'consumer poll timeout has expired' points at processing time. Heartbeat failures point at session timeout, GC pauses, or network stalls.

Confirm on the broker

kafka-log-dirs.sh --bootstrap-server localhost:9092 --describe --topic-list orders | head -40

Interpretation

Group coordinator logs and partition sizes show whether the churn coincides with a broker restart, leader election, or a single hot partition slowing one consumer.

Root cause

A Kafka consumer must call poll() often enough to stay in the group. When processing a batch takes longer than max.poll.interval.ms, the client stops sending heartbeats, the coordinator removes it, and every remaining member revokes and reacquires partitions. If the workload repeats, each cycle triggers another generation, and offsets committed by the departed member may be replayed.

Fix

Lower max.poll.records so a batch fits comfortably inside max.poll.interval.ms, or raise max.poll.interval.ms above the worst-case batch time. Move slow work off the poll thread, use static group membership with group.instance.id to survive short restarts, and prefer the cooperative-sticky assignor to reduce stop-the-world revocations. Keep session.timeout.ms above expected GC and network pauses.

Risk note

Raising max.poll.interval.ms delays detection of a genuinely stuck consumer and leaves partitions unassigned longer. Static membership without unique instance IDs can leave a zombie member holding partitions. Cooperative rebalancing still pauses affected partitions briefly.

Verify the fix

Run the group for at least three expected rebalance windows and confirm the generation number stays constant while lag decreases. Check consumer logs for the absence of poll timeout messages and verify no duplicate processing in downstream idempotency metrics.

A rebalance is a group-wide pause

Every time membership changes, Kafka revokes partitions from consumers and reassigns them. During that window nobody consumes the affected partitions. A “storm” is the same cycle repeating because the consumer that triggered it rejoins and times out again.

[Consumer clientId=orders-3, groupId=orders] Rebalance began
[Consumer clientId=orders-3, groupId=orders] Revoking previously assigned partitions
[Consumer clientId=orders-3, groupId=orders] Sending LeaveGroup request ... due to consumer poll timeout has expired

The last line is the giveaway: processing exceeded max.poll.interval.ms.

Measure the churn

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group orders --state --members --verbose

Run the describe twice, a minute apart. A generation that keeps climbing, or partition ownership moving without a deployment, confirms the loop. Then grep the consumer logs for the timeout message and correlate the time with batch size and processing latency.

Break the cycle

Reduce max.poll.records so the largest batch finishes well within the interval, or increase max.poll.interval.ms when records are genuinely slow. Move blocking calls to a worker pool and pause/resume partitions instead of holding the poll loop. Add group.instance.id for static membership so a rolling restart does not force a full rebalance, and use cooperative-sticky to revoke only the partitions that must move.

max.poll.records=100
max.poll.interval.ms=600000
session.timeout.ms=60000
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
group.instance.id=orders-pod-3

Verify stability

Watch the group for several rebalance windows. --state should report Stable continuously, the generation should stop changing, and lag should trend down. After a deliberate pod restart with static membership, confirm the member rejoins within the session timeout without revoking every partition. Finally, check downstream deduplication counters to ensure the storm did not leave duplicate side effects.