Topics & Partitions
BeginnerA topic is a named log of events; partitions are ordered, immutable sequences that allow parallelism — more partitions mean more consumer throughput but higher metadata overhead.
Overview
Topics are the fundamental unit of data organisation in Kafka. Think of a topic as a named category or feed — producers write records to a topic, and consumers subscribe to read from it. Unlike a traditional queue where a message is consumed once and deleted, Kafka topics retain all records for a configurable retention period, allowing multiple independent consumer groups to read the same data at their own pace. Topics are physically divided into partitions: ordered, immutable, append-only sequences of records. Each record within a partition has a unique, monotonically increasing offset. Partitions are the unit of parallelism — a topic with 12 partitions can be consumed by up to 12 consumers within a single consumer group simultaneously. The partition count is set at topic creation and is non-trivial to change later, making initial sizing decisions important.
Topic Structure & Partition Layout
A topic is spread across multiple brokers — each partition is stored on exactly one broker (its leader) and replicated to others (followers). The partition leader handles all reads and writes; followers replicate passively.
Record ordering is only guaranteed within a partition, not across them. If you need global ordering for a specific entity (e.g., all events for a given user), use a message key — Kafka routes records with the same key to the same partition via consistent hashing.
// Create a topic programmatically using AdminClient
Properties adminProps = new Properties();
adminProps.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
try (AdminClient admin = AdminClient.create(adminProps)) {
int numPartitions = 12;
short replicationFactor = 3;
NewTopic ordersTopic = new NewTopic("orders", numPartitions, replicationFactor);
// Optional: set topic-level config
ordersTopic.configs(Map.of(
"retention.ms", String.valueOf(7 * 24 * 60 * 60 * 1000L), // 7 days
"compression.type","snappy"
));
admin.createTopics(List.of(ordersTopic)).all().get();
System.out.println("Topic created successfully");
}Partition Assignment & Message Keys
When a producer sends a record: • If a key is provided → partition = murmur2(key) % numPartitions (same key always goes to the same partition, preserving order per key). • If no key → round-robin or sticky partition strategy (configurable via partitioner.class).
Key-based partitioning is critical for ordering guarantees — e.g., route all events for orderId=123 to the same partition so a single consumer always processes them in sequence.
// Producer — send with a key to guarantee ordering per entity
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
String orderId = "ORD-4321"; // key → always same partition
String payload = "{"status":"SHIPPED"}";
ProducerRecord<String, String> record =
new ProducerRecord<>("orders", orderId, payload);
producer.send(record, (metadata, ex) -> {
if (ex == null) {
System.out.printf("Sent to partition=%d offset=%d%n",
metadata.partition(), metadata.offset());
}
});
}Choosing the Right Partition Count
Too few partitions limit throughput and consumer parallelism. Too many partitions increase broker memory, replication overhead, and end-to-end latency (more files to fsync).
A common rule of thumb: target throughput / throughput per partition. For most use cases, start with max(target_consumers, throughput_MB_s / 10) and adjust based on benchmarks. Remember: you can increase partitions later but cannot decrease them without recreating the topic.
# Describe topic to inspect partition layout
kafka-topics.sh --bootstrap-server localhost:9092 \
--describe --topic orders
# Output:
# Topic: orders PartitionCount: 12 ReplicationFactor: 3
# Topic: orders Partition: 0 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3
# Topic: orders Partition: 1 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1
# ...
# Increase partitions (WARNING: breaks key-based ordering for existing records)
kafka-topics.sh --bootstrap-server localhost:9092 \
--alter --topic orders --partitions 24Key Points to Remember
- 1A topic is a named, durable, append-only log; partitions are its physical subdivisions, each stored on a single broker leader.
- 2Ordering is guaranteed only within a partition — use message keys to ensure all events for the same entity land in the same partition.
- 3The number of partitions is the maximum degree of parallelism for a consumer group; you cannot have more active consumers than partitions.
- 4Kafka retains records regardless of consumption — multiple consumer groups can read the same topic independently at their own offsets.
- 5Each record in a partition has a unique, monotonically increasing offset; there are no global offsets across partitions.
- 6Partition count is non-trivial to change after creation; increasing it breaks key-based ordering for keys that haven't been produced yet.
Interview Questions
Sign in to ask AriaWhy does Kafka use partitions and what problem do they solve?
How does Kafka determine which partition a message goes to when a key is provided?
What are the trade-offs of having too many or too few partitions in a Kafka topic?
If a Kafka topic has 6 partitions and a consumer group has 8 consumers, what happens?
You need to guarantee that all events for a given customer are processed in order. How do you achieve this in Kafka?
Ask Aria about Topics & Partitions
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.