Cheat SheetsApache KafkaProducers

Producers — Cheat Sheet

Apache Kafka · 7 topics. Download the PDF or the Instagram carousel and share it.

Cheat Sheet · AiCanCode.org

Producers

Apache Kafka7 topicsQuick revision reference

Kafka Producer Basics

A producer sends records to a topic; it serialises the key and value, selects a partition (by key hash, custom partitioner, or round-robin), and batches records for efficiency.

✓Records with the same key always go to the same partition — guarantees per-key ordering.
✓Records without a key use sticky partitioning (Kafka 2.4+) for better batching.
✓batch.size + linger.ms control how long the producer waits to fill a batch before sending.
✓enable.idempotence=true prevents duplicate records on producer retry (requires acks=all).
✓Transactional producers enable atomic writes across multiple topics/partitions.
✓Compression (zstd/lz4) reduces network and disk I/O — always use for high-throughput topics.

Properties + Java — Spring Boot producer config

# application.properties — producer config
spring.kafka.bootstrap-servers=localhost:9092

spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.springframework.kafka.support.serializer.JsonSerializer
spring.kafka.producer.properties.spring.json.add.type.headers=false

# Durability
spring.kafka.producer.acks=all
spring.kafka.producer.properties.enable.idempotence=true

# Throughput tuning
spring.kafka.producer.batch-size=65536          # 64 KB batch
spring.kafka.producer.properties.linger.ms=20   # wait up to 20 ms to fill batch
spring.kafka.producer.compression-type=zstd     # compress batches

# Retry
spring.kafka.producer.retries=3
spring.kafka.producer.properties.retry.backoff.ms=500

// KafkaTemplate usage
@Service
public class EventPublisher {
    private final KafkaTemplate<String, OrderEvent> kafka;

    public void publish(OrderEvent event) {
        CompletableFuture<SendResult<String, OrderEvent>> future =
            kafka.send("orders", event.getOrderId(), event);

        future.whenComplete((result, ex) -> {
            if (ex != null) log.error("Send failed", ex);
            else log.info("Sent to {}-{} @ offset {}",
                result.getRecordMetadata().topic(),
                result.getRecordMetadata().partition(),
                result.getRecordMetadata().offset());
        });
    }
}

Producer Acknowledgements (acks)

acks=0 (fire-and-forget), acks=1 (leader ack), acks=all (all ISR ack) — higher acks improve durability at the cost of latency; combine with min.insync.replicas for safety.

✓acks=0 (fire-and-forget), acks=1 (leader only), acks=all (all ISR) — higher acks = higher durability, higher latency.
✓acks=all alone is not sufficient if the ISR contains only one replica; combine with min.insync.replicas=2 for genuine durability.
✓The recommended production combination for critical data: replication.factor=3, min.insync.replicas=2, acks=all.
✓enable.idempotence=true automatically sets acks=all, retries=MAX_INT, and max.in.flight.requests.per.connection=5.
✓If min.insync.replicas is not satisfied, the broker throws NotEnoughReplicasException — this is a safety guard, not a bug.
✓For non-critical high-throughput data (analytics, metrics), acks=1 or acks=0 is acceptable to maximise throughput.

Java — Kafka Producer Config

// Producer configuration examples
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,   StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

// --- acks=0: fire-and-forget (metrics, non-critical logs) ---
props.put(ProducerConfig.ACKS_CONFIG, "0");
props.put(ProducerConfig.RETRIES_CONFIG, "0");  // no point retrying — no response anyway

// --- acks=1: leader ack only (moderate durability) ---
props.put(ProducerConfig.ACKS_CONFIG, "1");

// --- acks=all: strongest durability (financial, order events) ---
props.put(ProducerConfig.ACKS_CONFIG, "all");   // or "-1"
props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE + "");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true"); // auto-sets acks=all + retries

KafkaProducer<String, String> producer = new KafkaProducer<>(props);

Message Keys & Partitioning Strategy

Records with the same key are routed to the same partition, preserving order per entity; null keys distribute records round-robin across partitions.

✓Kafka ordering guarantee is per-partition only — use the entity ID as the key for per-entity ordering.
✓Default partitioner: murmur2(key) % numPartitions — deterministic and consistent for the same key.
✓Null keys use round-robin (or sticky batching in Kafka 2.4+) — no ordering guarantee across records.
✓Low-cardinality keys (status, region, boolean) cause hot partitions — use high-cardinality keys like entity IDs.
✓Co-partitioning is required for KStream-KTable joins: both topics must have the same key and partition count.
✓Changing partition count after topic creation breaks key → partition mapping — plan partition count upfront.

Java — key-based partition routing

// Order events keyed by orderId — all events for order-42 go to same partition
// Consumer sees: ORDER_CREATED → PAYMENT_RECEIVED → ORDER_SHIPPED (in order)
producer.send(new ProducerRecord<>(
    "order-events",
    order.getId().toString(),   // key = orderId → partition selection
    orderEvent                  // value
));

// Spring Kafka — specify key
kafkaTemplate.send("order-events", order.getId().toString(), orderEvent);

// Verify which partition a key maps to (useful for debugging)
int numPartitions = 12;
int partition = Utils.toPositive(Utils.murmur2("order-42".getBytes()))
                % numPartitions;
// → deterministic partition for any given key

// NULL key: round-robin (or sticky batch since Kafka 2.4)
// Use for events with no ordering requirement (metrics, logs)
kafkaTemplate.send("analytics-events", null, clickEvent);

// Ordering within partition is per-producer:
// If 2 producer instances send for same key, ordering is NOT guaranteed
// across instances — use sticky partitioner per producer for batching

Idempotent Producer

enable.idempotence=true assigns a producer ID and sequence number to each record; the broker deduplicates retries, guaranteeing exactly-once delivery per partition.

✓enable.idempotence=true assigns each producer a PID and adds per-partition sequence numbers
✓Broker deduplicates retried batches by checking the PID+partition sequence window
✓Idempotence forces acks=all, retries=MAX_INT, max.in.flight≤5 — set automatically
✓PID is ephemeral: producer restart gets a new PID, resetting the deduplication window
✓Idempotent producer gives exactly-once per partition per session — not cross-session
✓Transactional producer builds on idempotence to add atomic multi-partition writes

Kafka — enable.idempotence config

# application.properties (Spring Kafka)
spring.kafka.producer.properties.enable.idempotence=true
# Implicitly sets:
#   acks=all
#   retries=Integer.MAX_VALUE
#   max.in.flight.requests.per.connection=5

# Java producer config
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker:9092");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
// The following are set automatically but can be overridden compatibly:
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);

KafkaProducer<String, String> producer = new KafkaProducer<>(props);

Transactional Producer

Transactions span multiple partitions atomically; beginTransaction/commitTransaction ensure consume-transform-produce pipelines are atomic without duplicates or losses.

✓Transactional producer requires transactional.id (unique, stable per instance) + initTransactions()
✓Transactions atomically write to multiple partitions + commit consumer offsets
✓ProducerFencedException means another producer with the same transactional.id has taken over
✓Consumers must set isolation.level=read_committed to skip aborted transaction records
✓LSO (last stable offset) stops advancing if a transaction is left open — monitor for consumer lag
✓Spring Kafka: use KafkaTransactionManager + @Transactional for declarative EOS

Kafka — transactional producer EOS pipeline

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker:9092");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "order-processor-0"); // stable, unique per instance

KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.initTransactions();    // fence any previous zombie with same transactional.id

try {
    producer.beginTransaction();
    // Atomically write to multiple partitions
    producer.send(new ProducerRecord<>("orders-processed", orderId, processedJson));
    producer.send(new ProducerRecord<>("inventory-reserved", itemId, reserveJson));
    // Atomically commit consumer offset as part of the same transaction
    producer.sendOffsetsToTransaction(
        Map.of(new TopicPartition("orders-raw", partition),
               new OffsetAndMetadata(offset + 1)),
        consumerGroupMetadata
    );
    producer.commitTransaction();
} catch (ProducerFencedException e) {
    producer.close();           // this instance has been fenced by a newer one
} catch (KafkaException e) {
    producer.abortTransaction(); // roll back — consumer offset not advanced
    // Retry from the unconsumed offset
}

Producer Batching & Compression

linger.ms and batch.size control how long the producer buffers records before sending; compression.type (snappy, lz4, zstd) reduces network and storage cost at the broker.

✓batch.size: max bytes per batch; linger.ms: max wait time — batch sends when either limit hit
✓linger.ms=0 sends each record immediately (lowest latency); linger.ms=10+ improves throughput
✓Compression applied at producer, stored at broker, decompressed at consumer — reduces I/O end-to-end
✓lz4 = fastest (low latency); zstd = best ratio (low storage); snappy = good balance
✓buffer.memory exhaustion → send() blocks for max.block.ms then throws TimeoutException
✓Monitor kafka_producer_records_per_request_avg — higher is better batching efficiency

Kafka — batch.size, linger.ms, compression config

# High-throughput producer config
spring.kafka.producer.properties.batch.size=65536          # 64 KB batch (default 16KB)
spring.kafka.producer.properties.linger.ms=10             # wait up to 10ms to fill batch
spring.kafka.producer.properties.compression.type=lz4     # fast compression
spring.kafka.producer.properties.buffer.memory=67108864   # 64 MB total producer buffer
spring.kafka.producer.properties.acks=1                   # leader ack (throughput over durability)

# Low-latency producer (e.g. real-time event streaming)
spring.kafka.producer.properties.linger.ms=0              # send immediately
spring.kafka.producer.properties.batch.size=16384         # default — each record sent alone
spring.kafka.producer.properties.acks=1

# Trade-offs table:
# linger.ms=0,  batch=16KB  → lowest latency, lowest throughput (1 record/request)
# linger.ms=5,  batch=64KB  → good balance for most use cases
# linger.ms=20, batch=512KB → highest throughput, 20ms added latency

Serializers & Schema Registry

Serializers convert Java objects to bytes; the Confluent Schema Registry enforces Avro/JSON/Protobuf schemas and enables schema evolution with backward/forward compatibility.

✓Schema Registry stores versioned schemas; the schema ID (4 bytes) is prepended to every Avro/JSON Schema message.
✓BACKWARD compatibility (default): add fields with defaults, remove optional fields — consumers on old schema still work.
✓FULL_TRANSITIVE is the safest setting: enforces both backward and forward compatibility against all historical versions.
✓Avro is more efficient (binary, no field names in each message) than JSON but requires the avsc file and codegen.
✓For Spring Kafka without Schema Registry, use JsonSerializer + trusted.packages to prevent deserialization attacks.
✓Schema evolution in Avro must add new fields with "default" values — fields without defaults are breaking changes.

Avro + Java — Avro schema + producer setup

// order-event.avsc
{
  "type": "record",
  "name": "OrderEvent",
  "namespace": "com.example.events",
  "fields": [
    {"name": "orderId",    "type": "string"},
    {"name": "customerId", "type": "string"},
    {"name": "amount",     "type": "double"},
    {"name": "status",     "type": {"type": "enum",
                                    "name": "OrderStatus",
                                    "symbols": ["PENDING","CONFIRMED","SHIPPED"]}}
  ]
}

# pom.xml — Avro codegen plugin
<plugin>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro-maven-plugin</artifactId>
  <version>1.11.3</version>
  <executions>
    <execution>
      <goals><goal>schema</goal></goals>
      <configuration>
        <sourceDirectory>src/main/avro</sourceDirectory>
      </configuration>
    </execution>
  </executions>
</plugin>

// Producer config with Schema Registry
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
          KafkaAvroSerializer.class);
props.put("schema.registry.url", "http://schema-registry:8081");

// KafkaAvroSerializer auto-registers schema and prepends schema ID (4 bytes) to message
producer.send(new ProducerRecord<>("order-events", event.getOrderId().toString(), avroEvent));

Learn this free with Aria, your AI tutor → AiCanCode.org/learn/kafka

Cheat Sheet

Producers

Apache Kafka

7 topics · swipe →

AiCanCode.org

Producers

Kafka Producer Basics

✓Records with the same key always go to the same partition — guarantees per-key ordering.
✓Records without a key use sticky partitioning (Kafka 2.4+) for better batching.
✓batch.size + linger.ms control how long the producer waits to fill a batch before sending.
✓enable.idempotence=true prevents duplicate records on producer retry (requires acks=all).
✓Transactional producers enable atomic writes across multiple topics/partitions.

# application.properties — producer config
spring.kafka.bootstrap-servers=localhost:9092

spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.springframework.kafka.support.serializer.JsonSerializer
spring.kafka.producer.properties.spring.json.add.type.headers=false

# Durability
spring.kafka.producer.acks=all
…

AiCanCode.org

Producers

Producer Acknowledgements (acks)

✓acks=0 (fire-and-forget), acks=1 (leader only), acks=all (all ISR) — higher acks = higher durability, higher latency.
✓acks=all alone is not sufficient if the ISR contains only one replica; combine with min.insync.replicas=2 for genuine durability.
✓The recommended production combination for critical data: replication.factor=3, min.insync.replicas=2, acks=all.
✓enable.idempotence=true automatically sets acks=all, retries=MAX_INT, and max.in.flight.requests.per.connection=5.
✓If min.insync.replicas is not satisfied, the broker throws NotEnoughReplicasException — this is a safety guard, not a bug.

// Producer configuration examples
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,   StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

// --- acks=0: fire-and-forget (metrics, non-critical logs) ---
props.put(ProducerConfig.ACKS_CONFIG, "0");
props.put(ProducerConfig.RETRIES_CONFIG, "0");  // no point retrying — no response anyway
…

AiCanCode.org

Producers

Message Keys & Partitioning Strategy

✓Kafka ordering guarantee is per-partition only — use the entity ID as the key for per-entity ordering.
✓Default partitioner: murmur2(key) % numPartitions — deterministic and consistent for the same key.
✓Null keys use round-robin (or sticky batching in Kafka 2.4+) — no ordering guarantee across records.
✓Low-cardinality keys (status, region, boolean) cause hot partitions — use high-cardinality keys like entity IDs.
✓Co-partitioning is required for KStream-KTable joins: both topics must have the same key and partition count.

// Order events keyed by orderId — all events for order-42 go to same partition
// Consumer sees: ORDER_CREATED → PAYMENT_RECEIVED → ORDER_SHIPPED (in order)
producer.send(new ProducerRecord<>(
    "order-events",
    order.getId().toString(),   // key = orderId → partition selection
    orderEvent                  // value
));

// Spring Kafka — specify key
…

AiCanCode.org

Producers

Idempotent Producer

✓enable.idempotence=true assigns each producer a PID and adds per-partition sequence numbers
✓Broker deduplicates retried batches by checking the PID+partition sequence window
✓Idempotence forces acks=all, retries=MAX_INT, max.in.flight≤5 — set automatically
✓PID is ephemeral: producer restart gets a new PID, resetting the deduplication window
✓Idempotent producer gives exactly-once per partition per session — not cross-session

# application.properties (Spring Kafka)
spring.kafka.producer.properties.enable.idempotence=true
# Implicitly sets:
#   acks=all
#   retries=Integer.MAX_VALUE
#   max.in.flight.requests.per.connection=5

# Java producer config
Properties props = new Properties();
…

AiCanCode.org

Producers

Transactional Producer

✓Transactional producer requires transactional.id (unique, stable per instance) + initTransactions()
✓Transactions atomically write to multiple partitions + commit consumer offsets
✓ProducerFencedException means another producer with the same transactional.id has taken over
✓Consumers must set isolation.level=read_committed to skip aborted transaction records
✓LSO (last stable offset) stops advancing if a transaction is left open — monitor for consumer lag

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker:9092");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "order-processor-0"); // stable, unique per instance

KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.initTransactions();    // fence any previous zombie with same transactional.id

try {
…

AiCanCode.org

Producers

Producer Batching & Compression

✓batch.size: max bytes per batch; linger.ms: max wait time — batch sends when either limit hit
✓linger.ms=0 sends each record immediately (lowest latency); linger.ms=10+ improves throughput
✓Compression applied at producer, stored at broker, decompressed at consumer — reduces I/O end-to-end
✓lz4 = fastest (low latency); zstd = best ratio (low storage); snappy = good balance
✓buffer.memory exhaustion → send() blocks for max.block.ms then throws TimeoutException

# High-throughput producer config
spring.kafka.producer.properties.batch.size=65536          # 64 KB batch (default 16KB)
spring.kafka.producer.properties.linger.ms=10             # wait up to 10ms to fill batch
spring.kafka.producer.properties.compression.type=lz4     # fast compression
spring.kafka.producer.properties.buffer.memory=67108864   # 64 MB total producer buffer
spring.kafka.producer.properties.acks=1                   # leader ack (throughput over durability)

# Low-latency producer (e.g. real-time event streaming)
spring.kafka.producer.properties.linger.ms=0              # send immediately
…

AiCanCode.org

Producers

Serializers & Schema Registry

✓Schema Registry stores versioned schemas; the schema ID (4 bytes) is prepended to every Avro/JSON Schema message.
✓BACKWARD compatibility (default): add fields with defaults, remove optional fields — consumers on old schema still work.
✓FULL_TRANSITIVE is the safest setting: enforces both backward and forward compatibility against all historical versions.
✓Avro is more efficient (binary, no field names in each message) than JSON but requires the avsc file and codegen.
✓For Spring Kafka without Schema Registry, use JsonSerializer + trusted.packages to prevent deserialization attacks.

// order-event.avsc
{
  "type": "record",
  "name": "OrderEvent",
  "namespace": "com.example.events",
  "fields": [
    {"name": "orderId",    "type": "string"},
    {"name": "customerId", "type": "string"},
    {"name": "amount",     "type": "double"},
…

AiCanCode.org

Master Producers

Full lessons + an AI tutor, free.

AiCanCode.org

/learn/kafka