Log Retention & Compaction
BeginnerKafka retains messages on disk regardless of consumer acknowledgement. Retention can be time-based, size-based, or compaction-based, giving full control over data lifetime.
Overview
Unlike traditional message brokers that delete messages after consumption, Kafka keeps all messages in an append-only, immutable log. Retention policies determine when old data is removed. Log compaction keeps only the latest value per key, enabling Kafka to serve as a durable key-value store.
Time & Size Based Retention
retention.ms (default 7 days) deletes log segments older than the threshold. retention.bytes caps total partition size. Both can be applied together — whichever triggers first wins.
# Set retention on an existing topic
kafka-configs.sh --bootstrap-server localhost:9092 \
--entity-type topics --entity-name orders \
--alter \
--add-config retention.ms=86400000,retention.bytes=1073741824Log Compaction
Compaction keeps the latest message for each key. Tombstones (null-value messages) mark deletes. Set cleanup.policy=compact for state topics.
kafka-topics.sh --bootstrap-server localhost:9092 \
--create --topic user-profiles \
--partitions 6 --replication-factor 3 \
--config cleanup.policy=compact \
--config min.cleanable.dirty.ratio=0.5
// Tombstone — delete a key
ProducerRecord<String, String> tombstone =
new ProducerRecord<>("user-profiles", "user-123", null);
producer.send(tombstone);Key Points to Remember
- 1Kafka retains messages after consumption — consumers are independent
- 2retention.ms and retention.bytes control time/size-based deletion
- 3Log compaction keeps only the latest value per key
- 4Tombstones (null values) signal key deletion in compacted topics
- 5cleanup.policy=compact,delete combines both strategies
Interview Questions
Sign in to ask AriaWhat is the difference between time-based and size-based retention in Kafka?
How does log compaction work and when would you use it?
What is a tombstone message in Kafka and how does it trigger key deletion?
How can a Kafka topic act as a durable key-value store using compaction?
What happens to a compacted topic if a producer sends null value for a key?
Ask Aria about Log Retention & Compaction
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.