Message Keys & Partitioning Strategy
IntermediateRecords with the same key are routed to the same partition, preserving order per entity; null keys distribute records round-robin across partitions.
Overview
Kafka guarantees ordering only within a single partition. The message key determines which partition a record lands in: the default partitioner applies murmur2 hash(key) % numPartitions. All records with the same key go to the same partition, guaranteeing that events for a given entity (e.g. orderId=42) are processed in order by a single consumer thread. Records with null keys are distributed round-robin. Partitioning strategy has deep implications for consumer parallelism, hot partitions, and co-partitioning requirements for Kafka Streams joins. A custom partitioner overrides the default routing logic for use cases like routing premium customers to dedicated partitions.
Key-based partitioning for ordering guarantees
Use the entity ID as the key to ensure all events for one entity are ordered within a partition. Consumer threads process one partition each, maintaining per-entity order.
// Order events keyed by orderId — all events for order-42 go to same partition
// Consumer sees: ORDER_CREATED → PAYMENT_RECEIVED → ORDER_SHIPPED (in order)
producer.send(new ProducerRecord<>(
"order-events",
order.getId().toString(), // key = orderId → partition selection
orderEvent // value
));
// Spring Kafka — specify key
kafkaTemplate.send("order-events", order.getId().toString(), orderEvent);
// Verify which partition a key maps to (useful for debugging)
int numPartitions = 12;
int partition = Utils.toPositive(Utils.murmur2("order-42".getBytes()))
% numPartitions;
// → deterministic partition for any given key
// NULL key: round-robin (or sticky batch since Kafka 2.4)
// Use for events with no ordering requirement (metrics, logs)
kafkaTemplate.send("analytics-events", null, clickEvent);
// Ordering within partition is per-producer:
// If 2 producer instances send for same key, ordering is NOT guaranteed
// across instances — use sticky partitioner per producer for batchingHot partitions and key cardinality
A skewed key distribution causes hot partitions — one partition receiving disproportionate traffic, becoming a throughput bottleneck. High-cardinality keys distribute evenly.
// BAD: low cardinality key → hot partition
// 90% of orders are status='PENDING' → 90% of traffic on one partition
producer.send(new ProducerRecord<>("order-events",
order.getStatus(), // ← poor key: only 3–5 unique values
orderEvent));
// GOOD: high cardinality key → even distribution
producer.send(new ProducerRecord<>("order-events",
order.getId().toString(), // ← millions of unique IDs
orderEvent));
// Diagnose hot partitions with Kafka metrics
// kafka.server:type=BrokerTopicMetrics,topic=order-events,name=MessagesInPerSec
// If one partition has 10x the messages of others → hot partition
// Fix skewed keys by adding a suffix
String key = order.getCustomerId() + "-" + (System.nanoTime() % 10);
// Trade-off: loses strict per-customer ordering
// Count distribution across partitions
kafka-run-class.sh kafka.tools.GetOffsetShell \
--broker-list localhost:9092 \
--topic order-events \
--time -1 # latest offset per partition
# If one partition has significantly higher offset than others → skewCustom partitioner for priority routing
Implement Partitioner to override default key hashing — useful for routing premium customers to dedicated partitions or controlling data locality.
// Custom partitioner: premium customers → partitions 0–2, standard → 3–11
public class PriorityPartitioner implements Partitioner {
@Override
public int partition(String topic, Object key, byte[] keyBytes,
Object value, byte[] valueBytes, Cluster cluster) {
int numPartitions = cluster.partitionCountForTopic(topic);
int premiumPartitions = Math.min(3, numPartitions);
int standardPartitions = numPartitions - premiumPartitions;
if (value instanceof OrderEvent order && order.isPremium()) {
// Route premium orders to first 3 partitions
return Math.abs(key.hashCode()) % premiumPartitions;
}
// Standard orders to remaining partitions
return premiumPartitions + (Math.abs(key.hashCode()) % standardPartitions);
}
@Override public void close() {}
@Override public void configure(Map<String, ?> configs) {}
}
// Register custom partitioner
props.put(ProducerConfig.PARTITIONER_CLASS_CONFIG,
PriorityPartitioner.class.getName());
// WARNING: custom partition logic must be consistent —
// changing it requires a full topic rewrite to maintain orderingKey Points to Remember
- 1Kafka ordering guarantee is per-partition only — use the entity ID as the key for per-entity ordering.
- 2Default partitioner: murmur2(key) % numPartitions — deterministic and consistent for the same key.
- 3Null keys use round-robin (or sticky batching in Kafka 2.4+) — no ordering guarantee across records.
- 4Low-cardinality keys (status, region, boolean) cause hot partitions — use high-cardinality keys like entity IDs.
- 5Co-partitioning is required for KStream-KTable joins: both topics must have the same key and partition count.
- 6Changing partition count after topic creation breaks key → partition mapping — plan partition count upfront.
Interview Questions
Sign in to ask AriaHow does Kafka guarantee ordering for events related to the same entity?
What is a hot partition and how do you diagnose and fix it?
If two producer instances send events for the same key, is global ordering guaranteed?
What is co-partitioning and why is it required for Kafka Streams joins?
What happens to key → partition mapping if you increase the partition count of an existing topic?
Ask Aria about Message Keys & Partitioning Strategy
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.