Home/Learn/Apache Kafka/Retention Policies

Retention Policies

Intermediate

Administration

retention.ms and retention.bytes control time- and size-based log retention; set per-topic based on replay requirements and storage budget.

Overview

Kafka retains messages on disk regardless of whether consumers have read them. Retention is controlled at the topic level with two independent limits: time-based (retention.ms) and size-based (retention.bytes). When either limit is exceeded, the oldest log segments are deleted. Alternatively, log.cleanup.policy=compact enables log compaction — instead of deleting old segments, Kafka keeps only the latest message per key, making the log act like a key-value store (useful for CDC changelog topics and consumer group offset topics). Tiered storage (Kafka 3.6+) extends retention cheaply by offloading old segments to object storage (S3) while keeping recent segments local for low-latency reads.

Configuring retention per topic

Retention settings can be set at broker level (defaults) or overridden per topic. The kafka-configs.sh tool or AdminClient API modifies topic configs at runtime without downtime.

Shell — topic retention configuration

# Broker defaults (server.properties)
log.retention.ms=604800000      # 7 days (default)
log.retention.bytes=-1          # unlimited by size (default)
log.segment.bytes=1073741824    # 1 GB per log segment file
log.segment.ms=604800000        # roll segment after 7 days even if not full

# Create topic with custom retention
kafka-topics.sh --bootstrap-server localhost:9092   --create --topic order-events   --partitions 12 --replication-factor 3   --config retention.ms=86400000     # 24 hours
  --config retention.bytes=10737418240  # 10 GB max

# Override retention on existing topic
kafka-configs.sh --bootstrap-server localhost:9092   --entity-type topics --entity-name order-events   --alter   --add-config retention.ms=172800000  # change to 48 hours

# Verify
kafka-configs.sh --bootstrap-server localhost:9092   --entity-type topics --entity-name order-events   --describe

Log compaction for changelog topics

Compacted topics retain the latest value per key indefinitely (or until the key is tombstoned with a null value). They are used for CDC, materialised views, and consumer group offset storage (__consumer_offsets).

Shell + Java — log compaction setup

# Create a compacted topic (e.g. for user profile CDC)
kafka-topics.sh --bootstrap-server localhost:9092   --create --topic user-profiles   --partitions 6 --replication-factor 3   --config cleanup.policy=compact   --config min.cleanable.dirty.ratio=0.1   --config segment.ms=3600000     # compact hourly

# Key compaction settings
# min.cleanable.dirty.ratio: lower = more aggressive compaction
# delete.retention.ms: how long tombstone (null-value) records are kept (default 24h)
# min.compaction.lag.ms: minimum time before a message can be compacted

# Publish a tombstone to delete a key from the compacted log
ProducerRecord<String, String> tombstone =
    new ProducerRecord<>("user-profiles", "user-123", null); // null value = delete
producer.send(tombstone);

# Mixed policy: delete old + compacted for recent
kafka-configs.sh ... --add-config "cleanup.policy=[compact,delete]"

Spring Boot: AdminClient for runtime topic management

Spring's KafkaAdmin bean creates topics defined as @Bean TopicBuilder instances at startup. AdminClient enables programmatic inspection and modification of topic configs.

Java — Spring KafkaAdmin topic declaration

@Configuration
public class KafkaTopicConfig {

    @Bean
    public KafkaAdmin kafkaAdmin(KafkaProperties props) {
        return new KafkaAdmin(Map.of(
            AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG,
            props.getBootstrapServers()
        ));
    }

    // Spring creates this topic on startup if it doesn't exist
    @Bean
    public NewTopic orderEventsTopic() {
        return TopicBuilder.name("order-events")
            .partitions(12)
            .replicas(3)
            .config(TopicConfig.RETENTION_MS_CONFIG, "86400000")   // 24h
            .config(TopicConfig.RETENTION_BYTES_CONFIG, "5368709120") // 5 GB
            .build();
    }

    @Bean
    public NewTopic userProfilesTopic() {
        return TopicBuilder.name("user-profiles")
            .partitions(6)
            .replicas(3)
            .config(TopicConfig.CLEANUP_POLICY_CONFIG,
                    TopicConfig.CLEANUP_POLICY_COMPACT)
            .build();
    }
}

Key Points to Remember

1Retention is segment-based — old segments are deleted as a unit when the oldest message in them expires.
2retention.ms and retention.bytes are independent; either can trigger deletion; -1 disables that dimension.
3Log compaction retains only the latest message per key — use it for CDC and materialised-view topics.
4A null-value message (tombstone) marks a key for deletion from a compacted topic; it is retained for delete.retention.ms before removal.
5Segment size affects deletion granularity — a 1 GB segment means up to 1 GB of messages older than retention.ms may be kept until the segment rolls.
6For long retention (months/years), consider Tiered Storage (Kafka 3.6+) to offload cold segments to S3 cheaply.

Interview Questions

What is the difference between log deletion and log compaction retention policies?

EasyConfluent

How do you delete a specific key from a compacted Kafka topic?

MediumNetflix

Why does retention.ms not guarantee exact time-based deletion and what controls the granularity?

HardAmazon

What is the cleanup.policy=compact,delete combination and when would you use it?

MediumLinkedIn

How would you implement an event replay capability for a new microservice joining the system 6 months later?

MediumUber

Ask Aria about Retention Policies

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

Log Compaction

Topic Configuration & Tuning