Tiered Storage
AdvancedKafka Tiered Storage offloads older log segments to object storage (S3, GCS) while keeping recent data on local broker disks, dramatically reducing storage cost for long-retention topics.
Overview
Recent data stays on fast local SSD for low-latency reads; older segments are asynchronously uploaded to cheap object storage and fetched on demand. Brokers can have different local retention sizes while sharing the remote history.
Enabling Tiered Storage (Kafka 3.6+)
Configure the remote log storage plugin and set local vs total retention separately.
# server.properties
remote.log.storage.system.enable=true
remote.log.storage.manager.class.name=\
org.apache.kafka.server.log.remote.storage.RemoteLogStorageManager
# Topic-level — 1 day local, 30 days total (rest goes to S3)
kafka-topics.sh --create \
--bootstrap-server localhost:9092 \
--topic audit-logs \
--config remote.storage.enable=true \
--config local.retention.ms=86400000 \ # 1 day on local disk
--config retention.ms=2592000000 # 30 days totalKey Points to Remember
- 1Recent segments on local SSD; older segments in object storage
- 2local.retention.ms controls how long to keep data locally
- 3retention.ms is the total retention across local + remote
- 4Reduces broker disk cost by 60-80% for long-retention topics
- 5Remote reads have higher latency — only for historical data fetch
Interview Questions
Sign in to ask AriaWhat problem does Kafka Tiered Storage solve?
What is the difference between local.retention.ms and retention.ms in a tiered storage topic?
How does tiered storage affect read latency for recent vs historical consumers?
What are the cost implications of tiered storage vs increasing broker disk?
When would tiered storage NOT be a good fit for a Kafka topic?
Ask Aria about Tiered Storage
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.