Home/Learn/Apache Kafka/Kafka Performance Tuning

Kafka Performance Tuning

Advanced

Throughput levers: batch.size, linger.ms, compression, partition count; latency levers: acks, buffer.memory, fetch.min.bytes; monitor under-replicated partitions and GC pauses.

Overview

Kafka performance tuning addresses two often-conflicting objectives: maximum throughput and minimum latency. Throughput is maximised by batching more records (larger batch.size, non-zero linger.ms), enabling compression, and increasing partition count. Latency is minimised by reducing linger.ms to 0, using acks=1 (or even acks=0 for fire-and-forget), and tuning consumer fetch parameters to return data sooner. The right settings depend on whether your use case is a bulk data pipeline (favour throughput) or an interactive request path (favour latency). Monitor the key health signals: under-replicated partition count, consumer group lag, broker network throughput, and JVM GC pause duration.

Producer throughput tuning

The producer accumulates records into batches per partition before sending. batch.size (default 16 KB) sets the maximum batch size; linger.ms (default 0) adds an artificial delay to allow more records to accumulate before sending. Larger batches and non-zero linger.ms dramatically improve throughput. Enable compression (lz4 for speed, zstd for best ratio) to reduce network and disk usage.

YAML — producer throughput tuning configuration

# Producer throughput config (application.yml / producer properties)
spring:
  kafka:
    producer:
      batch-size: 65536          # 64 KB (default 16 KB) — larger batches
      buffer-memory: 67108864    # 64 MB total memory buffer
      compression-type: lz4      # lz4=speed, zstd=ratio, snappy=balanced
      acks: all                  # durability; use acks=1 for max throughput
      properties:
        linger.ms: 20            # wait 20ms to accumulate batch (0=no wait)
        max.in.flight.requests.per.connection: 5

# Tuning summary:
# linger.ms=0 + small batch.size → low latency, low throughput
# linger.ms=20+ + large batch.size → high throughput, higher latency
# compression saves ~50-80% network bandwidth for text/JSON payloads
# max.in.flight=5 with enable.idempotence=true → safe pipelining

Consumer throughput and latency tuning

fetch.min.bytes causes the broker to wait until at least N bytes are available before responding to a fetch request — this batches reads and increases throughput at the cost of added latency. fetch.max.wait.ms caps how long the broker waits. max.poll.records limits how many records are returned per poll() call, protecting against long processing cycles that cause session timeouts.

YAML — consumer fetch tuning for throughput vs latency trade-offs

# Consumer fetch config
spring:
  kafka:
    consumer:
      fetch-min-size: 65536       # 64 KB — broker waits for this many bytes (throughput)
      fetch-max-wait: 500         # max 500ms broker wait even if < fetch.min.bytes
      max-poll-records: 500       # records per poll() call
      properties:
        max.partition.fetch.bytes: 1048576   # 1 MB per partition per fetch
        session.timeout.ms: 30000            # heartbeat timeout
        heartbeat.interval.ms: 10000         # 1/3 of session.timeout.ms

# Consumer scaling: partitions = maximum parallelism
# If topic has 12 partitions: max 12 consumers in same group
# Add more partitions if throughput bottleneck (cannot reduce without rebalance)

# Low-latency consumer (minimal buffering)
fetch-min-size: 1           # return immediately with any available data
fetch-max-wait: 0           # no waiting
max-poll-records: 10        # small batches → more frequent processing

Partition count, compression, and monitoring

Partition count is the primary throughput lever — more partitions = more parallelism. But partitions have costs: each open file handle on the broker, more leader election overhead, and longer rebalance times. A rule of thumb: target 1 MB/s per partition and aim for partition count = peak throughput / (single partition throughput). Monitor under-replicated partitions (should be 0) and consumer group lag as the key health signals.

Notes + CLI — partition sizing, compression choice, and monitoring metrics

# Partition count tuning formula
# Target: 100 MB/s total topic throughput
# Single partition throughput: ~50 MB/s on well-specced broker
# Required partitions: 100/50 = 2 minimum
# Add buffer for growth: 6 partitions

# Compression benchmark (relative performance on JSON payloads)
# none:    baseline throughput, max CPU, max disk
# snappy:  2x throughput, moderate CPU, 40% disk reduction
# lz4:     3x throughput, low CPU, 45% disk reduction  ← best for throughput
# zstd:    2.5x throughput, moderate CPU, 60% disk reduction  ← best ratio
# gzip:    1.5x throughput, high CPU, 65% disk reduction

# Key Prometheus metrics to monitor
# under_replicated_partitions        → must be 0 in production
# kafka_consumer_group_lag           → upstream processing bottleneck
# kafka_network_io_bytes_total       → broker network saturation
# kafka_server_replica_fetcher_lag   → replica falling behind leader
# jvm_gc_pause_seconds_max           → GC stop-the-world hurting broker latency

# Inspect consumer lag (CLI)
kafka-consumer-groups.sh --bootstrap-server broker:9092 \
  --describe --group order-service-group

Key Points to Remember

1Throughput levers: increase batch.size, set linger.ms > 0, enable compression (lz4 for speed, zstd for ratio)
2Latency levers: set linger.ms=0, fetch-min-size=1, fetch-max-wait=0, small max-poll-records
3Partition count is the primary parallelism lever — more partitions = more consumer parallelism, but more broker overhead
4Consumer group lag is the most important operational metric — it shows whether consumers can keep up with producers
5Under-replicated partitions should always be 0 in production — non-zero indicates a broker or network problem
6JVM GC pauses affect broker latency — use G1GC/ZGC with tuned heap and avoid throughput GC on brokers

Interview Questions

What is the role of linger.ms and batch.size in Kafka producer throughput?

MediumConfluent

How do you choose between lz4, snappy, gzip, and zstd compression for a Kafka topic?

MediumLinkedIn

A Kafka topic has 3 partitions but you need to process 10 MB/s. What are your options?

HardUber

What is consumer group lag and what does it indicate about system health?

EasyAmazon

How does fetch.min.bytes improve consumer throughput and what is the latency trade-off?

MediumNetflix

Ask Aria about Kafka Performance Tuning

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

MirrorMaker 2 for Geo-Replication

Kafka Design Patterns