Cheat SheetsApache KafkaAdvanced

Advanced — Cheat Sheet

Apache Kafka · 8 topics. Download the PDF or the Instagram carousel and share it.

Cheat Sheet · AiCanCode.org

Advanced

Apache Kafka8 topicsQuick revision reference

Message Compression

Producer-side compression (snappy, lz4, zstd, gzip) compresses record batches; the broker stores them compressed and consumers decompress, reducing I/O and storage significantly.

✓Compression is applied per batch on the producer — larger batches produce better compression ratios
✓End-to-end compression: producer compresses → broker stores compressed → consumer auto-decompresses (no broker CPU cost)
✓zstd is recommended for new deployments: best ratio/CPU balance; lz4 for maximum throughput on CPU-constrained producers
✓compression.type=producer on the broker means "store as received" — the default and most efficient setting
✓compression-rate-avg JMX metric shows effectiveness (0.3 = 70% size reduction); monitor to verify savings
✓JSON and Avro payloads compress well (40–80% reduction); binary or already-compressed payloads compress poorly

YAML + CLI — producer compression configuration and codec comparison

# application.yml — producer compression
spring:
  kafka:
    producer:
      compression-type: zstd   # none | gzip | snappy | lz4 | zstd
      # Compression is applied per batch — larger batches compress better
      batch-size: 65536        # 64 KB (default 16 KB)
      properties:
        linger.ms: 20          # wait 20ms to accumulate batch before sending

# Compression ratio benchmarks (JSON payloads, approximate)
# none:    1.0x ratio, 0 CPU
# snappy:  2.5x ratio, low CPU    — good for CPU-constrained producers
# lz4:     3.0x ratio, very low CPU — best throughput
# zstd:    4.0x ratio, moderate CPU — best ratio/CPU balance
# gzip:    4.5x ratio, high CPU   — best compression, highest cost

# Per-topic compression override (broker-level)
# compression.type=producer means: use whatever the producer sent (default)
# compression.type=gzip means: broker recompresses with gzip (additional CPU)

kafka-configs.sh --bootstrap-server broker:9092 \
  --entity-type topics --entity-name my-topic \
  --alter --add-config compression.type=producer

Kafka Cluster Scaling

Add brokers and use partition reassignment to rebalance leadership; Kafka 3.x's Cruise Control automates load balancing based on disk, CPU, and network metrics.

✓Adding brokers does not auto-rebalance — use kafka-reassign-partitions.sh or Cruise Control to move partitions
✓Always throttle partition reassignment to prevent overwhelming brokers and impacting live traffic
✓Partition count increase is irreversible and breaks key ordering — plan partition counts upfront
✓Preferred leader election restores balanced leadership after broker restarts or completed reassignments
✓Cruise Control automates rebalancing decisions based on disk, CPU, and network utilisation metrics
✓In Strimzi (Kubernetes), Cruise Control is integrated — trigger rebalances via KafkaRebalance CRDs

CLI — partition reassignment after adding a broker, with throttle

# Step 1: Generate reassignment plan for a topic
cat > topics-to-move.json <<EOF
{"topics": [{"topic": "orders"}], "version": 1}
EOF

kafka-reassign-partitions.sh \
  --bootstrap-server broker:9092 \
  --broker-list "1,2,3,4"   ← new broker 4 is added \
  --topics-to-move-json-file topics-to-move.json \
  --generate > reassignment-plan.json

# Step 2: Execute with throttle (limit to 50 MB/s to avoid impacting live traffic)
kafka-reassign-partitions.sh \
  --bootstrap-server broker:9092 \
  --reassignment-json-file reassignment-plan.json \
  --execute \
  --throttle 52428800    # 50 MB/s in bytes

# Step 3: Monitor progress
kafka-reassign-partitions.sh \
  --bootstrap-server broker:9092 \
  --reassignment-json-file reassignment-plan.json \
  --verify

# Step 4: Remove throttle after reassignment completes
kafka-configs.sh --bootstrap-server broker:9092 \
  --entity-type brokers --entity-default \
  --alter --delete-config leader.replication.throttled.rate

MirrorMaker 2 for Geo-Replication

MirrorMaker 2 (MM2) is a Kafka Connect-based tool that replicates topics across clusters for disaster recovery, data locality, or multi-region active-active deployments.

✓MM2 is built on Kafka Connect — run it as a Connect worker with mirror connector configs.
✓MirrorSourceConnector replicates data; MirrorCheckpointConnector translates offsets; MirrorHeartbeatConnector measures lag.
✓Replicated topics are prefixed with source alias (us-east.orders) to prevent cycles.
✓Translated offsets enable consumers to resume near their original position after failover.
✓Active-active requires both directions enabled; cycle prevention is automatic via prefix policy.
✓sync.group.offsets.enabled keeps consumer group positions in sync for near-seamless failover.

Properties — MirrorMaker 2 configuration

# mm2.properties — standalone or Connect distributed worker config
clusters = us-east, eu-west

us-east.bootstrap.servers = kafka-us-east:9092
eu-west.bootstrap.servers = kafka-eu-west:9092

# Replicate from us-east to eu-west
us-east->eu-west.enabled = true
us-east->eu-west.topics = orders, payments, inventory.*

# Offset and heartbeat replication
us-east->eu-west.emit.checkpoints.enabled = true
us-east->eu-west.emit.heartbeats.enabled  = true
us-east->eu-west.sync.group.offsets.enabled = true

# Replication factor on the target cluster
replication.factor = 3

# Topic renaming — "orders" on us-east becomes "us-east.orders" on eu-west
# (override with replication.policy.class for custom naming)

Kafka Performance Tuning

Throughput levers: batch.size, linger.ms, compression, partition count; latency levers: acks, buffer.memory, fetch.min.bytes; monitor under-replicated partitions and GC pauses.

✓Throughput levers: increase batch.size, set linger.ms > 0, enable compression (lz4 for speed, zstd for ratio)
✓Latency levers: set linger.ms=0, fetch-min-size=1, fetch-max-wait=0, small max-poll-records
✓Partition count is the primary parallelism lever — more partitions = more consumer parallelism, but more broker overhead
✓Consumer group lag is the most important operational metric — it shows whether consumers can keep up with producers
✓Under-replicated partitions should always be 0 in production — non-zero indicates a broker or network problem
✓JVM GC pauses affect broker latency — use G1GC/ZGC with tuned heap and avoid throughput GC on brokers

YAML — producer throughput tuning configuration

# Producer throughput config (application.yml / producer properties)
spring:
  kafka:
    producer:
      batch-size: 65536          # 64 KB (default 16 KB) — larger batches
      buffer-memory: 67108864    # 64 MB total memory buffer
      compression-type: lz4      # lz4=speed, zstd=ratio, snappy=balanced
      acks: all                  # durability; use acks=1 for max throughput
      properties:
        linger.ms: 20            # wait 20ms to accumulate batch (0=no wait)
        max.in.flight.requests.per.connection: 5

# Tuning summary:
# linger.ms=0 + small batch.size → low latency, low throughput
# linger.ms=20+ + large batch.size → high throughput, higher latency
# compression saves ~50-80% network bandwidth for text/JSON payloads
# max.in.flight=5 with enable.idempotence=true → safe pipelining

Kafka Design Patterns

Event notification, event-carried state transfer, event sourcing, CQRS with Kafka, the outbox pattern, and the inbox pattern are common Kafka-based architectural patterns.

✓Event Notification: thin event with ID; consumers call back — simple but coupled
✓Event-Carried State Transfer: full state in event; consumers autonomous — preferred for projections
✓Outbox Pattern: write state + event in one DB transaction; relay publishes to Kafka — no dual-write
✓Inbox Pattern: store received events in local table for idempotent, ordered processing
✓CQRS: commands update write model; events update multiple independent read projections
✓Debezium CDC is the most reliable Outbox relay — zero polling latency, no app-level poller needed

Kafka — Event Notification vs Event-Carried State Transfer

// Event Notification — thin event (ID only)
public record OrderPlacedEvent(String orderId, Instant occurredAt) {}
// Consumer receives orderId, calls back: GET /orders/{orderId}
// ❌ Coupling: consumer fails if order-service is down

// Event-Carried State Transfer — full state in event
public record OrderPlacedEvent(
    String orderId,
    String customerId,
    List<OrderLine> lines,
    BigDecimal total,
    String currency,
    Address shippingAddress,
    Instant occurredAt
) {}
// Consumer is fully autonomous — no callback needed
// ✓ Resilient: works even if order-service is temporarily down
// ✓ Replayable: projections rebuilt from past events include all data

// When to prefer notification:
// - Event payload would be huge and rarely needed
// - Data is sensitive (PII) and should not be in the event log
// - Low-volume events where a callback is acceptable

Multi-Datacenter Replication (MirrorMaker 2)

MirrorMaker 2 replicates Kafka topics across clusters and datacenters for disaster recovery, geo-replication, and data aggregation pipelines.

✓MM2 mirrors topics with source prefix (us-east.orders) to prevent loops
✓MM2 mirrors consumer group offsets for transparent failover
✓Active-passive: one cluster is primary, the other is DR standby
✓Active-active: bidirectional — requires careful loop prevention
✓CheckpointConnector translates remote offsets for seamless consumer failover

MirrorMaker 2 — active-passive replication

# mm2.properties
clusters=us-east, eu-west
us-east.bootstrap.servers=kafka-us-east:9092
eu-west.bootstrap.servers=kafka-eu-west:9092

# Replicate us-east → eu-west (active-passive)
us-east->eu-west.enabled=true
us-east->eu-west.topics=orders.*,payments.*
us-east->eu-west.groups=.*       # mirror consumer group offsets

replication.factor=3
tasks.max=4

# Start MM2
connect-mirror-maker.sh mm2.properties

Kafka Performance Tuning Playbook

Kafka throughput and latency are tunable via producer batching, compression, consumer fetch settings, and broker I/O configuration. Understanding the trade-offs lets you hit SLA targets.

✓linger.ms > 0 batches messages for better throughput at cost of latency
✓LZ4 compression gives the best Kafka throughput per CPU cycle
✓fetch.min.bytes reduces consumer poll frequency → higher throughput
✓More partitions = more parallelism, but more file handles and rebalance cost
✓Use kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh to benchmark

Kafka — performance tuning settings

# High-throughput producer
linger.ms=20                       # wait 20ms to fill batch
batch.size=131072                  # 128 KB (default 16 KB)
compression.type=lz4               # fast + good ratio
buffer.memory=67108864             # 64 MB send buffer
max.in.flight.requests.per.connection=5

# Low-latency producer
linger.ms=0                        # send immediately
compression.type=none
acks=1

# Consumer throughput tuning
fetch.min.bytes=65536              # 64 KB before return (default 1B)
fetch.max.wait.ms=500
max.poll.records=1000
max.partition.fetch.bytes=10485760 # 10 MB per partition

# Benchmark
kafka-producer-perf-test.sh --topic test --num-records 1000000 \
  --record-size 1024 --throughput -1 --bootstrap-server localhost:9092

Tiered Storage

Kafka Tiered Storage offloads older log segments to object storage (S3, GCS) while keeping recent data on local broker disks, dramatically reducing storage cost for long-retention topics.

✓Recent segments on local SSD; older segments in object storage
✓local.retention.ms controls how long to keep data locally
✓retention.ms is the total retention across local + remote
✓Reduces broker disk cost by 60-80% for long-retention topics
✓Remote reads have higher latency — only for historical data fetch

Kafka — tiered storage configuration

# server.properties
remote.log.storage.system.enable=true
remote.log.storage.manager.class.name=\
  org.apache.kafka.server.log.remote.storage.RemoteLogStorageManager

# Topic-level — 1 day local, 30 days total (rest goes to S3)
kafka-topics.sh --create \
  --bootstrap-server localhost:9092 \
  --topic audit-logs \
  --config remote.storage.enable=true \
  --config local.retention.ms=86400000    \ # 1 day on local disk
  --config retention.ms=2592000000           # 30 days total

Learn this free with Aria, your AI tutor → AiCanCode.org/learn/kafka

Cheat Sheet

Advanced

Apache Kafka

8 topics · swipe →

AiCanCode.org

Advanced

Message Compression

✓Compression is applied per batch on the producer — larger batches produce better compression ratios
✓End-to-end compression: producer compresses → broker stores compressed → consumer auto-decompresses (no broker CPU cost)
✓zstd is recommended for new deployments: best ratio/CPU balance; lz4 for maximum throughput on CPU-constrained producers
✓compression.type=producer on the broker means "store as received" — the default and most efficient setting
✓compression-rate-avg JMX metric shows effectiveness (0.3 = 70% size reduction); monitor to verify savings

# application.yml — producer compression
spring:
  kafka:
    producer:
      compression-type: zstd   # none | gzip | snappy | lz4 | zstd
      # Compression is applied per batch — larger batches compress better
      batch-size: 65536        # 64 KB (default 16 KB)
      properties:
        linger.ms: 20          # wait 20ms to accumulate batch before sending
…

AiCanCode.org

Advanced

Kafka Cluster Scaling

✓Adding brokers does not auto-rebalance — use kafka-reassign-partitions.sh or Cruise Control to move partitions
✓Always throttle partition reassignment to prevent overwhelming brokers and impacting live traffic
✓Partition count increase is irreversible and breaks key ordering — plan partition counts upfront
✓Preferred leader election restores balanced leadership after broker restarts or completed reassignments
✓Cruise Control automates rebalancing decisions based on disk, CPU, and network utilisation metrics

# Step 1: Generate reassignment plan for a topic
cat > topics-to-move.json <<EOF
{"topics": [{"topic": "orders"}], "version": 1}
EOF

kafka-reassign-partitions.sh \
  --bootstrap-server broker:9092 \
  --broker-list "1,2,3,4"   ← new broker 4 is added \
  --topics-to-move-json-file topics-to-move.json \
…

AiCanCode.org

Advanced

MirrorMaker 2 for Geo-Replication

✓MM2 is built on Kafka Connect — run it as a Connect worker with mirror connector configs.
✓MirrorSourceConnector replicates data; MirrorCheckpointConnector translates offsets; MirrorHeartbeatConnector measures lag.
✓Replicated topics are prefixed with source alias (us-east.orders) to prevent cycles.
✓Translated offsets enable consumers to resume near their original position after failover.
✓Active-active requires both directions enabled; cycle prevention is automatic via prefix policy.

# mm2.properties — standalone or Connect distributed worker config
clusters = us-east, eu-west

us-east.bootstrap.servers = kafka-us-east:9092
eu-west.bootstrap.servers = kafka-eu-west:9092

# Replicate from us-east to eu-west
us-east->eu-west.enabled = true
us-east->eu-west.topics = orders, payments, inventory.*
…

AiCanCode.org

Advanced

Kafka Performance Tuning

✓Throughput levers: increase batch.size, set linger.ms > 0, enable compression (lz4 for speed, zstd for ratio)
✓Latency levers: set linger.ms=0, fetch-min-size=1, fetch-max-wait=0, small max-poll-records
✓Partition count is the primary parallelism lever — more partitions = more consumer parallelism, but more broker overhead
✓Consumer group lag is the most important operational metric — it shows whether consumers can keep up with producers
✓Under-replicated partitions should always be 0 in production — non-zero indicates a broker or network problem

# Producer throughput config (application.yml / producer properties)
spring:
  kafka:
    producer:
      batch-size: 65536          # 64 KB (default 16 KB) — larger batches
      buffer-memory: 67108864    # 64 MB total memory buffer
      compression-type: lz4      # lz4=speed, zstd=ratio, snappy=balanced
      acks: all                  # durability; use acks=1 for max throughput
      properties:
…

AiCanCode.org

Advanced

Kafka Design Patterns

✓Event Notification: thin event with ID; consumers call back — simple but coupled
✓Event-Carried State Transfer: full state in event; consumers autonomous — preferred for projections
✓Outbox Pattern: write state + event in one DB transaction; relay publishes to Kafka — no dual-write
✓Inbox Pattern: store received events in local table for idempotent, ordered processing
✓CQRS: commands update write model; events update multiple independent read projections

// Event Notification — thin event (ID only)
public record OrderPlacedEvent(String orderId, Instant occurredAt) {}
// Consumer receives orderId, calls back: GET /orders/{orderId}
// ❌ Coupling: consumer fails if order-service is down

// Event-Carried State Transfer — full state in event
public record OrderPlacedEvent(
    String orderId,
    String customerId,
…

AiCanCode.org

Advanced

Multi-Datacenter Replication (MirrorMaker 2)

✓MM2 mirrors topics with source prefix (us-east.orders) to prevent loops
✓MM2 mirrors consumer group offsets for transparent failover
✓Active-passive: one cluster is primary, the other is DR standby
✓Active-active: bidirectional — requires careful loop prevention
✓CheckpointConnector translates remote offsets for seamless consumer failover

# mm2.properties
clusters=us-east, eu-west
us-east.bootstrap.servers=kafka-us-east:9092
eu-west.bootstrap.servers=kafka-eu-west:9092

# Replicate us-east → eu-west (active-passive)
us-east->eu-west.enabled=true
us-east->eu-west.topics=orders.*,payments.*
us-east->eu-west.groups=.*       # mirror consumer group offsets
…

AiCanCode.org

Advanced

Kafka Performance Tuning Playbook

✓linger.ms > 0 batches messages for better throughput at cost of latency
✓LZ4 compression gives the best Kafka throughput per CPU cycle
✓fetch.min.bytes reduces consumer poll frequency → higher throughput
✓More partitions = more parallelism, but more file handles and rebalance cost
✓Use kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh to benchmark

# High-throughput producer
linger.ms=20                       # wait 20ms to fill batch
batch.size=131072                  # 128 KB (default 16 KB)
compression.type=lz4               # fast + good ratio
buffer.memory=67108864             # 64 MB send buffer
max.in.flight.requests.per.connection=5

# Low-latency producer
linger.ms=0                        # send immediately
…

AiCanCode.org

Advanced

Tiered Storage

✓Recent segments on local SSD; older segments in object storage
✓local.retention.ms controls how long to keep data locally
✓retention.ms is the total retention across local + remote
✓Reduces broker disk cost by 60-80% for long-retention topics
✓Remote reads have higher latency — only for historical data fetch

# server.properties
remote.log.storage.system.enable=true
remote.log.storage.manager.class.name=\
  org.apache.kafka.server.log.remote.storage.RemoteLogStorageManager

# Topic-level — 1 day local, 30 days total (rest goes to S3)
kafka-topics.sh --create \
  --bootstrap-server localhost:9092 \
  --topic audit-logs \
…

AiCanCode.org

Master Advanced

Full lessons + an AI tutor, free.

AiCanCode.org

/learn/kafka