Kafka producer TimeoutException — Expiring Records in Batch

Diagnose Kafka producer timeouts by checking topic metadata, ISR health, and delivery timeout budgets, then fix acks, linger, retries, and broker capacity.

advanceddebuggingkafkaproducer

Symptoms

org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for orders-0: 30000 ms has passed since batch creation
Topic orders not present in metadata after 60000 ms
Produce requests succeed in development but time out under load
Producer throughput collapses while consumer lag is normal

Possible causes

delivery.timeout.ms is shorter than linger.ms plus request.timeout.ms plus retry backoff
acks=all cannot be satisfied because the ISR has fewer replicas than min.insync.replicas
Brokers are unreachable, overloaded, or have a full disk, so requests never complete
A producer creates a new client per request, so batches never fill and metadata is fetched repeatedly

Diagnosis steps

Confirm the topic exists and has leaders

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders

Interpretation

A missing topic or a partition with no leader explains metadata timeouts. Check that the leader and ISR lists contain live brokers.

Check broker-level under-replicated partitions

kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-replicated-partitions

Interpretation

Partitions listed here have fewer in-sync replicas. With acks=all and min.insync.replicas=2, a single-replica ISR makes every produce fail with NotEnoughReplicas or timeout.

Produce a message with explicit acks and timing

kafka-console-producer.sh --bootstrap-server localhost:9092 --topic orders --producer-property acks=all --producer-property delivery.timeout.ms=10000 <<< 'test-order'

Interpretation

An immediate error isolates broker-side rejection; a hang until the timeout points to network reachability, metadata, or request queue saturation.

Measure end-to-end produce latency

kafka-producer-perf-test.sh --topic orders --num-records 1000 --record-size 200 --throughput 100 --producer-props bootstrap.servers=localhost:9092 acks=all

Interpretation

Compare the reported average and max latency with delivery.timeout.ms. High max latency with low throughput indicates broker throttling or retry amplification.

Root cause

A Kafka producer batches records per partition and only sends when linger.ms or batch.size is reached. Each record must be acknowledged within delivery.timeout.ms; that budget covers waiting for the batch, the request, and all retries. If the ISR cannot satisfy acks=all, or the broker is slow, the record expires before it is acknowledged, and the client raises TimeoutException even though the broker may still have appended an earlier attempt.

Fix

Set delivery.timeout.ms comfortably above linger.ms + request.timeout.ms + retries * retry.backoff.ms, reduce linger.ms for latency-sensitive traffic, and keep acks=all with a healthy ISR. Restore broker capacity or disk, fix under-replicated partitions, and reuse a single long-lived producer. Enable idempotence so retries do not duplicate records.

Risk note

Lowering acks to 1 improves availability but can lose acknowledged records during a leader failure. Retrying a timed-out batch can duplicate messages unless idempotence is enabled. Increasing delivery.timeout.ms makes failures slower to surface and can increase memory held by the accumulator.

Verify the fix

Run the perf test again with the production acks setting and confirm max latency stays below delivery.timeout.ms with zero expired records. Check under-replicated partitions is empty and producer metrics record-s-error stays at zero during a broker restart.

What “expiring” actually means

The producer does not send each record immediately. It appends to a per-partition batch and waits for linger.ms or batch.size, then sends. A record must be acknowledged before delivery.timeout.ms expires, and that single budget includes batch wait, request time, and every retry.

org.apache.kafka.common.errors.TimeoutException:
Expiring 1 record(s) for orders-0: 30000 ms has passed since batch creation

The broker may have received the first attempt; the client simply never saw a successful response in time. That ambiguity is why idempotence matters.

Check metadata and ISR first

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders
kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-replicated-partitions

If the topic has no leader, or the ISR has dropped below min.insync.replicas, acks=all can never succeed. Fix broker health and replication before touching producer settings. A disk-full broker often shows up exactly this way.

Make the timeout budget consistent

acks=all
enable.idempotence=true
linger.ms=20
request.timeout.ms=30000
delivery.timeout.ms=120000
retries=2147483647

delivery.timeout.ms must exceed linger plus request timeout plus the retry window. Keep producers long-lived and share them across requests; a producer per request defeats batching and refetches metadata constantly.

Verify under failure

Run a perf test with acks=all, then restart one broker and watch producer metrics. record-error-rate should return to zero, record-queue-time-max should fall, and no records should expire. Confirm under-replicated partitions recovers and that consumers see each order exactly once, proving idempotent retries did their job.