Producer Batching & Compression
Intermediatelinger.ms and batch.size control how long the producer buffers records before sending; compression.type (snappy, lz4, zstd) reduces network and storage cost at the broker.
Overview
Kafka producers achieve high throughput by **batching** multiple records into a single network request before sending to the broker. Two settings control batch formation: `batch.size` (max bytes per batch per partition — default 16 KB) and `linger.ms` (max time to wait for more records to fill the batch — default 0 ms, send immediately). Setting `linger.ms = 5–20` and `batch.size = 65536` trades a small latency increase for significant throughput improvement. **Compression** (`compression.type`: `snappy`, `lz4`, `zstd`, `gzip`) is applied per batch at the producer — the broker stores compressed batches and consumers decompress. `zstd` gives the best ratio; `lz4` gives the best speed. The combination of batching + compression is the primary lever for high-throughput Kafka producers.
batch.size and linger.ms — The Core Throughput Knobs
`batch.size` caps the batch in bytes; the batch is sent when it is full OR when `linger.ms` elapses — whichever comes first. With `linger.ms=0` (default), the producer sends immediately after receiving a record — effectively no batching for low-rate producers. `buffer.memory` is the total bytes the producer can buffer across all partitions before blocking.
# High-throughput producer config
spring.kafka.producer.properties.batch.size=65536 # 64 KB batch (default 16KB)
spring.kafka.producer.properties.linger.ms=10 # wait up to 10ms to fill batch
spring.kafka.producer.properties.compression.type=lz4 # fast compression
spring.kafka.producer.properties.buffer.memory=67108864 # 64 MB total producer buffer
spring.kafka.producer.properties.acks=1 # leader ack (throughput over durability)
# Low-latency producer (e.g. real-time event streaming)
spring.kafka.producer.properties.linger.ms=0 # send immediately
spring.kafka.producer.properties.batch.size=16384 # default — each record sent alone
spring.kafka.producer.properties.acks=1
# Trade-offs table:
# linger.ms=0, batch=16KB → lowest latency, lowest throughput (1 record/request)
# linger.ms=5, batch=64KB → good balance for most use cases
# linger.ms=20, batch=512KB → highest throughput, 20ms added latencyCompression Algorithms: Trade-offs
All four compression algorithms work on the record batch. `snappy` and `lz4` are CPU-cheap with moderate compression ratios — ideal for high-throughput pipelines. `zstd` (Kafka 2.1+) gives the best ratio with modest CPU overhead — best for storage-constrained topics. `gzip` is slowest and rarely used for new deployments. Compression happens at the producer; the broker stores compressed bytes; the consumer decompresses — network and disk I/O savings are realised throughout.
# Compression type comparison (typical JSON payloads):
# gzip: 70% size reduction, highest CPU, slowest
# snappy: 50% reduction, low CPU, fast — good default for high throughput
# lz4: 50% reduction, very low CPU, fastest — best latency-sensitive pipelines
# zstd: 65% reduction, moderate CPU — best ratio for storage-heavy topics
# Verify compression is actually being used (per-topic stats)
kafka-log-dirs.sh --bootstrap-server broker:9092 --topic-list orders --describe | grep -i compress
# Kafka metrics for producer batching efficiency
# kafka_producer_batch_size_avg — average bytes per batch sent
# kafka_producer_records_per_request_avg — records per request (higher = better batching)
# kafka_producer_compression_rate_avg — 1.0 = no compression, 0.3 = 70% reduction
# Spring Boot Prometheus metrics (with micrometer-kafka)
spring.kafka.producer.properties.interceptor.classes=\
io.micrometer.kafka.instrumentation.ProducerInterceptormax.request.size and buffer.memory — Preventing Producer Backpressure
If the producer fills `buffer.memory` faster than the broker can consume batches (slow broker, high acks), the `send()` call blocks for `max.block.ms` then throws a `TimeoutException`. Monitor `kafka_producer_waiting_threads` and `buffer.available.bytes`. For very large messages, increase `max.request.size` on both producer and `message.max.bytes` on the broker/topic.
# Large message support
spring.kafka.producer.properties.max.request.size=5242880 # 5 MB max record
spring.kafka.producer.properties.buffer.memory=67108864 # 64 MB buffer
# Broker-side: also increase for large messages
# kafka-configs.sh --alter --entity-type topics --entity-name orders
# --add-config max.message.bytes=5242880
# Backpressure handling: max.block.ms
spring.kafka.producer.properties.max.block.ms=5000 # block up to 5s then throw
# Monitoring producer buffer pressure
# kafka_producer_buffer_available_bytes → alert if < 10% of buffer.memory
# kafka_producer_waiting_threads → threads blocked on send() = backpressure
- alert: KafkaProducerBufferPressure
expr: kafka_producer_buffer_available_bytes / kafka_producer_buffer_total_bytes < 0.1
for: 30s
annotations:
summary: "Kafka producer buffer >90% full — broker may be slow or acks too high"Key Points to Remember
- 1batch.size: max bytes per batch; linger.ms: max wait time — batch sends when either limit hit
- 2linger.ms=0 sends each record immediately (lowest latency); linger.ms=10+ improves throughput
- 3Compression applied at producer, stored at broker, decompressed at consumer — reduces I/O end-to-end
- 4lz4 = fastest (low latency); zstd = best ratio (low storage); snappy = good balance
- 5buffer.memory exhaustion → send() blocks for max.block.ms then throws TimeoutException
- 6Monitor kafka_producer_records_per_request_avg — higher is better batching efficiency
Interview Questions
Sign in to ask AriaWhat is the effect of setting linger.ms=0 vs linger.ms=20 on a Kafka producer?
Which compression algorithm would you choose for a latency-sensitive pipeline?
What happens when the Kafka producer fills its buffer.memory?
How does batching relate to compression efficiency in Kafka?
What metric would you alert on to detect Kafka producer backpressure?
Ask Aria about Producer Batching & Compression
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.