Kafka Performance Tuning Playbook
AdvancedKafka throughput and latency are tunable via producer batching, compression, consumer fetch settings, and broker I/O configuration. Understanding the trade-offs lets you hit SLA targets.
Overview
For high-throughput use cases, tune producers (linger.ms, batch.size, compression), consumers (fetch.min.bytes, max.poll.records), and brokers (num.io.threads). For low-latency, minimise batching.
Producer Throughput vs Latency
linger.ms and batch.size control batching. Higher values → better throughput, higher latency.
# High-throughput producer
linger.ms=20 # wait 20ms to fill batch
batch.size=131072 # 128 KB (default 16 KB)
compression.type=lz4 # fast + good ratio
buffer.memory=67108864 # 64 MB send buffer
max.in.flight.requests.per.connection=5
# Low-latency producer
linger.ms=0 # send immediately
compression.type=none
acks=1
# Consumer throughput tuning
fetch.min.bytes=65536 # 64 KB before return (default 1B)
fetch.max.wait.ms=500
max.poll.records=1000
max.partition.fetch.bytes=10485760 # 10 MB per partition
# Benchmark
kafka-producer-perf-test.sh --topic test --num-records 1000000 \
--record-size 1024 --throughput -1 --bootstrap-server localhost:9092Key Points to Remember
- 1linger.ms > 0 batches messages for better throughput at cost of latency
- 2LZ4 compression gives the best Kafka throughput per CPU cycle
- 3fetch.min.bytes reduces consumer poll frequency → higher throughput
- 4More partitions = more parallelism, but more file handles and rebalance cost
- 5Use kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh to benchmark
Interview Questions
Sign in to ask AriaWhat does linger.ms do and how does it affect throughput vs latency?
Which compression algorithm gives the best Kafka throughput?
How does fetch.min.bytes improve consumer throughput?
You need sub-10ms latency for a trading system using Kafka. What settings would you change?
How would you benchmark a Kafka cluster to determine maximum throughput?
Ask Aria about Kafka Performance Tuning Playbook
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.