Home/Learn/Apache Kafka/Topic Configuration & Tuning

Topic Configuration & Tuning

Intermediate

Administration

Key configs: replication.factor, min.insync.replicas, unclean.leader.election.enable, message.max.bytes, segment.ms — each has significant durability and performance implications.

Overview

Every Kafka topic has per-topic configuration properties that override broker-level defaults and control durability, retention, and throughput. These can be set at creation or altered live with kafka-configs.sh or the AdminClient API. The most critical configs for production form the durability triangle: replication.factor (how many copies of each partition), min.insync.replicas (minimum acks before a produce is committed), and producer acks=all. Getting this wrong is the leading cause of silent data loss. Retention configs (retention.ms, retention.bytes) control storage cost; segment configs (segment.ms, segment.bytes) affect compaction speed and log recovery time.

Durability: replication.factor and min.insync.replicas

replication.factor determines how many broker replicas store each partition. min.insync.replicas defines the minimum number of in-sync replicas that must acknowledge a write before success. The safe pattern for production: replication.factor=3, min.insync.replicas=2, producer acks=all — tolerates one broker failure without data loss. Setting min.insync.replicas=1 with acks=all is effectively acks=1.

CLI — topic creation and durability configuration

# Create topic with durability settings
kafka-topics.sh --bootstrap-server broker:9092 \
  --create --topic payments \
  --partitions 12 \
  --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config unclean.leader.election.enable=false

# Alter existing topic config
kafka-configs.sh --bootstrap-server broker:9092 \
  --entity-type topics --entity-name payments \
  --alter \
  --add-config min.insync.replicas=2,unclean.leader.election.enable=false

# Producer properties (application.yml)
# spring.kafka.producer.acks=all
# spring.kafka.producer.properties.enable.idempotence=true
# spring.kafka.producer.properties.retries=3

# Describe current topic configuration
kafka-configs.sh --bootstrap-server broker:9092 \
  --entity-type topics --entity-name payments --describe

Retention and cleanup policies

cleanup.policy=delete (default) removes segments older than retention.ms or larger than retention.bytes. cleanup.policy=compact retains only the latest value per key — ideal for changelog topics and event sourcing snapshots. Combine both: cleanup.policy=compact,delete to compact old data while also enforcing a time or size ceiling.

CLI — retention and log compaction policy configuration

# Retention by time (reduce to 1 day for high-volume topics)
kafka-configs.sh --bootstrap-server broker:9092 \
  --entity-type topics --entity-name user-events \
  --alter \
  --add-config retention.ms=86400000     # 1 day

# Retention by size (100 MB per partition)
  --add-config retention.bytes=104857600

# Log compaction — keep latest value per key (e.g. user profile state)
kafka-topics.sh --create --topic user-profiles \
  --partitions 6 --replication-factor 3 \
  --config cleanup.policy=compact \
  --config min.cleanable.dirty.ratio=0.5 \
  --config segment.ms=3600000

# Combined: compact + delete (retain compacted records AND max age)
  --config cleanup.policy=compact,delete \
  --config retention.ms=604800000        # 7 days

Segment tuning and message size limits

Segments are physical log files on disk. segment.bytes (default 1 GB) and segment.ms (default 7 days) control when a new segment rolls. Smaller segments speed up compaction and deletion but create more open file handles. message.max.bytes must be coordinated with broker replica.fetch.max.bytes and consumer max.partition.fetch.bytes.

CLI — segment size, message size, and coordinator settings

# Topic-level message and segment tuning
kafka-configs.sh --bootstrap-server broker:9092 \
  --entity-type topics --entity-name bulk-uploads \
  --alter \
  --add-config max.message.bytes=10485760     # 10 MB max message
  --add-config segment.bytes=268435456        # 256 MB segments
  --add-config segment.ms=3600000             # roll every hour

# Consumer must allow same large messages
# max.partition.fetch.bytes=10485760

# Broker defaults (server.properties)
# message.max.bytes=1048576           # 1 MB default
# replica.fetch.max.bytes=1048576     # must be >= message.max.bytes
# log.segment.bytes=1073741824        # 1 GB default
# log.retention.hours=168             # 7 days default

Key Points to Remember

1Safety durability pattern: replication.factor=3, min.insync.replicas=2, producer acks=all — tolerates 1 broker failure without data loss
2unclean.leader.election.enable=false prevents an out-of-sync replica from becoming leader and causing silent data loss
3cleanup.policy=delete removes old data by time/size; cleanup.policy=compact keeps only the latest value per message key
4max.message.bytes must be consistent across topic config, broker replica.fetch.max.bytes, and consumer max.partition.fetch.bytes
5Smaller segment.bytes speeds up compaction and log deletion but increases open file handle count on brokers
6kafka-configs.sh --describe shows per-topic config overrides and distinguishes them from broker-level defaults

Interview Questions

What is the safe durability config for a production Kafka topic that must not lose data on a single broker failure?

MediumLinkedIn

If min.insync.replicas=1 and producer acks=all, is this actually safe? Why or why not?

MediumConfluent

What is the difference between cleanup.policy=delete and cleanup.policy=compact?

EasyFlipkart

Why must max.message.bytes, replica.fetch.max.bytes, and max.partition.fetch.bytes be coordinated?

HardUber

What are the trade-offs of setting segment.bytes to a small value like 10 MB?

HardNetflix

Ask Aria about Topic Configuration & Tuning

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

Retention Policies

Kafka Security (TLS & SASL)