Kafka Producer Basics
BeginnerA producer sends records to a topic; it serialises the key and value, selects a partition (by key hash, custom partitioner, or round-robin), and batches records for efficiency.
Overview
The Kafka producer accumulates records in an in-memory buffer, groups them into batches by partition, and sends them to the appropriate broker. The key drives partition selection: records with the same key always go to the same partition, preserving per-key ordering. Without a key, the default partitioner distributes records round-robin (or sticky partitioning in Kafka 2.4+). Tuning batch.size, linger.ms, and compression.type directly impacts throughput. The idempotent producer (enable.idempotence=true) prevents duplicate writes on retries.
Producer Configuration in Spring Boot
Spring auto-configures KafkaTemplate from spring.kafka.producer.* properties. Key settings: serializers, acks, retries, batch size, and linger.
# application.properties — producer config
spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.springframework.kafka.support.serializer.JsonSerializer
spring.kafka.producer.properties.spring.json.add.type.headers=false
# Durability
spring.kafka.producer.acks=all
spring.kafka.producer.properties.enable.idempotence=true
# Throughput tuning
spring.kafka.producer.batch-size=65536 # 64 KB batch
spring.kafka.producer.properties.linger.ms=20 # wait up to 20 ms to fill batch
spring.kafka.producer.compression-type=zstd # compress batches
# Retry
spring.kafka.producer.retries=3
spring.kafka.producer.properties.retry.backoff.ms=500
// KafkaTemplate usage
@Service
public class EventPublisher {
private final KafkaTemplate<String, OrderEvent> kafka;
public void publish(OrderEvent event) {
CompletableFuture<SendResult<String, OrderEvent>> future =
kafka.send("orders", event.getOrderId(), event);
future.whenComplete((result, ex) -> {
if (ex != null) log.error("Send failed", ex);
else log.info("Sent to {}-{} @ offset {}",
result.getRecordMetadata().topic(),
result.getRecordMetadata().partition(),
result.getRecordMetadata().offset());
});
}
}Partition Selection & Custom Partitioner
Records with the same key always land in the same partition (hash-based). Records without a key use sticky partitioning (Kafka 2.4+). Implement Partitioner for custom routing logic, e.g. routing VIP orders to dedicated partitions.
// Custom partitioner — VIP orders get partition 0; others distributed normally
public class VipFirstPartitioner implements Partitioner {
@Override
public int partition(String topic, Object key, byte[] keyBytes,
Object value, byte[] valueBytes, Cluster cluster) {
int numPartitions = cluster.partitionCountForTopic(topic);
if (value instanceof OrderEvent event && event.isVip()) {
return 0; // dedicated partition for VIP orders
}
// Default: murmur2 hash of key
return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;
}
@Override public void close() {}
@Override public void configure(Map<String, ?> configs) {}
}
# Register custom partitioner
spring.kafka.producer.properties.partitioner.class=com.example.VipFirstPartitioner
// Key-based ordering guarantee:
// All records with key "customer-42" go to the same partition
// → events for customer-42 are processed IN ORDER by the consumer
kafka.send("orders", "customer-42", event);Idempotent & Transactional Producer
enable.idempotence=true assigns a Producer ID and sequence number to every record — the broker deduplicates retries. Transactional producers wrap multiple sends in an atomic transaction, ensuring all or nothing semantics across partitions.
# Idempotent producer — prevents duplicate records on network retry
spring.kafka.producer.properties.enable.idempotence=true
spring.kafka.producer.acks=all # required for idempotence
spring.kafka.producer.retries=2147483647 # effectively unlimited retries
spring.kafka.producer.properties.max.in.flight.requests.per.connection=5 # up to 5 in-flight
# Transactional producer — atomic multi-partition writes
spring.kafka.producer.transaction-id-prefix=order-tx-
// Transactional send
@Bean
public KafkaTransactionManager<String, OrderEvent> txManager(
ProducerFactory<String, OrderEvent> pf) {
return new KafkaTransactionManager<>(pf);
}
// Use @Transactional — Spring wraps the sends in a Kafka transaction
@Transactional("kafkaTransactionManager")
public void publishWithTransaction(OrderEvent event, InventoryEvent inv) {
kafka.send("orders", event.getOrderId(), event);
kafka.send("inventory", inv.getSku(), inv);
// Both sends committed atomically, or both aborted on exception
}Key Points to Remember
- 1Records with the same key always go to the same partition — guarantees per-key ordering.
- 2Records without a key use sticky partitioning (Kafka 2.4+) for better batching.
- 3batch.size + linger.ms control how long the producer waits to fill a batch before sending.
- 4enable.idempotence=true prevents duplicate records on producer retry (requires acks=all).
- 5Transactional producers enable atomic writes across multiple topics/partitions.
- 6Compression (zstd/lz4) reduces network and disk I/O — always use for high-throughput topics.
Interview Questions
Sign in to ask AriaHow does the Kafka producer decide which partition to send a record to?
What is linger.ms and how does it affect throughput vs latency?
What is an idempotent producer and how does it prevent duplicates?
What is the difference between acks=1 and acks=all?
How do transactional producers achieve exactly-once semantics?
Ask Aria about Kafka Producer Basics
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.