Cheat SheetsApache KafkaAdministration

Administration — Cheat Sheet

Apache Kafka · 8 topics. Download the PDF or the Instagram carousel and share it.

Cheat Sheet · AiCanCode.org

Administration

Apache Kafka8 topicsQuick revision reference

Log Compaction

Compacted topics retain only the latest value per key, acting as a change-log materialised view; deleted records are signalled with a null-value tombstone.

✓cleanup.policy=compact retains the latest value per key indefinitely; this is different from time/size-based retention which deletes old segments.
✓Compaction is asynchronous — the "head" of the log always contains all recent records; only the "tail" is compacted.
✓A tombstone (key + null value) signals deletion; it is retained for delete.retention.ms before being purged.
✓Log compaction is the backbone of Kafka Streams KTable state stores — it allows state to be rebuilt by replaying the compacted changelog.
✓Kafka Connect CDC pipelines (Debezium) use compacted topics so consumers can reconstruct current DB state by replaying from offset 0.
✓Compacted topics and time-based retention can be combined: cleanup.policy=compact,delete — compaction within the retention window, deletion after.

Shell — Compacted Topic Setup

# Create a compacted topic
kafka-topics.sh --bootstrap-server localhost:9092 \
    --create \
    --topic user-profiles \
    --partitions 6 \
    --replication-factor 3 \
    --config cleanup.policy=compact \
    --config min.insync.replicas=2 \
    --config min.compaction.lag.ms=60000  # records survive at least 60s before compaction

# Compaction behaviour example for key "user-123":
# Offset 0: key=user-123, value={"name":"Alice","email":"a@ex.com"}
# Offset 5: key=user-123, value={"name":"Alice","email":"alice@ex.com"}  ← UPDATE
# Offset 9: key=user-123, value={"name":"Alice Smith","email":"alice@ex.com"} ← UPDATE
#
# After compaction — log retains only:
# key=user-123, value={"name":"Alice Smith","email":"alice@ex.com"} (offset 9)
# Offsets 0 and 5 are discarded.

# Convert an existing topic to compacted:
kafka-configs.sh --bootstrap-server localhost:9092 \
    --entity-type topics --entity-name user-profiles \
    --alter --add-config cleanup.policy=compact

Retention Policies

retention.ms and retention.bytes control time- and size-based log retention; set per-topic based on replay requirements and storage budget.

✓Retention is segment-based — old segments are deleted as a unit when the oldest message in them expires.
✓retention.ms and retention.bytes are independent; either can trigger deletion; -1 disables that dimension.
✓Log compaction retains only the latest message per key — use it for CDC and materialised-view topics.
✓A null-value message (tombstone) marks a key for deletion from a compacted topic; it is retained for delete.retention.ms before removal.
✓Segment size affects deletion granularity — a 1 GB segment means up to 1 GB of messages older than retention.ms may be kept until the segment rolls.
✓For long retention (months/years), consider Tiered Storage (Kafka 3.6+) to offload cold segments to S3 cheaply.

Shell — topic retention configuration

# Broker defaults (server.properties)
log.retention.ms=604800000      # 7 days (default)
log.retention.bytes=-1          # unlimited by size (default)
log.segment.bytes=1073741824    # 1 GB per log segment file
log.segment.ms=604800000        # roll segment after 7 days even if not full

# Create topic with custom retention
kafka-topics.sh --bootstrap-server localhost:9092   --create --topic order-events   --partitions 12 --replication-factor 3   --config retention.ms=86400000     # 24 hours
  --config retention.bytes=10737418240  # 10 GB max

# Override retention on existing topic
kafka-configs.sh --bootstrap-server localhost:9092   --entity-type topics --entity-name order-events   --alter   --add-config retention.ms=172800000  # change to 48 hours

# Verify
kafka-configs.sh --bootstrap-server localhost:9092   --entity-type topics --entity-name order-events   --describe

Topic Configuration & Tuning

Key configs: replication.factor, min.insync.replicas, unclean.leader.election.enable, message.max.bytes, segment.ms — each has significant durability and performance implications.

✓Safety durability pattern: replication.factor=3, min.insync.replicas=2, producer acks=all — tolerates 1 broker failure without data loss
✓unclean.leader.election.enable=false prevents an out-of-sync replica from becoming leader and causing silent data loss
✓cleanup.policy=delete removes old data by time/size; cleanup.policy=compact keeps only the latest value per message key
✓max.message.bytes must be consistent across topic config, broker replica.fetch.max.bytes, and consumer max.partition.fetch.bytes
✓Smaller segment.bytes speeds up compaction and log deletion but increases open file handle count on brokers
✓kafka-configs.sh --describe shows per-topic config overrides and distinguishes them from broker-level defaults

CLI — topic creation and durability configuration

# Create topic with durability settings
kafka-topics.sh --bootstrap-server broker:9092 \
  --create --topic payments \
  --partitions 12 \
  --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config unclean.leader.election.enable=false

# Alter existing topic config
kafka-configs.sh --bootstrap-server broker:9092 \
  --entity-type topics --entity-name payments \
  --alter \
  --add-config min.insync.replicas=2,unclean.leader.election.enable=false

# Producer properties (application.yml)
# spring.kafka.producer.acks=all
# spring.kafka.producer.properties.enable.idempotence=true
# spring.kafka.producer.properties.retries=3

# Describe current topic configuration
kafka-configs.sh --bootstrap-server broker:9092 \
  --entity-type topics --entity-name payments --describe

Kafka Security (TLS & SASL)

Enable TLS for encryption in transit; use SASL/PLAIN, SASL/SCRAM, or SASL/OAUTHBEARER for authentication; ACLs control per-topic produce/consume permissions.

✓Kafka security = TLS (encryption) + SASL (authentication) + ACLs (authorisation) — all three work independently or together
✓SASL/PLAIN transmits credentials in plaintext — always pair with TLS (use SASL_SSL protocol, not SASL_PLAINTEXT)
✓SASL/SCRAM-SHA-256/512 stores salted credential hashes — safer than PLAIN; use for non-Kerberos environments
✓Consumer groups also need ACLs — READ permission on the consumer group is required alongside READ on the topic
✓super.users in server.properties bypass all ACL checks — restrict this list to broker inter-communication only
✓SASL/OAUTHBEARER + JWT allows integration with an existing OAuth2 identity provider (Keycloak, Okta)

Config + YAML — broker TLS and Spring Boot client SSL configuration

# server.properties — broker TLS configuration
listeners=PLAINTEXT://:9092,SSL://:9093
advertised.listeners=PLAINTEXT://broker1:9092,SSL://broker1:9093
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL

ssl.keystore.location=/etc/kafka/ssl/broker.keystore.jks
ssl.keystore.password=keystore-password
ssl.key.password=key-password
ssl.truststore.location=/etc/kafka/ssl/broker.truststore.jks
ssl.truststore.password=truststore-password
ssl.client.auth=none          # none=one-way TLS, required=mTLS

# application.yml — Spring Boot client TLS config
spring:
  kafka:
    properties:
      security.protocol: SSL
      ssl.truststore.location: classpath:ssl/client.truststore.jks
      ssl.truststore.password: ${KAFKA_TRUSTSTORE_PASSWORD}
      # For mTLS (client certificate)
      ssl.keystore.location: classpath:ssl/client.keystore.jks
      ssl.keystore.password: ${KAFKA_KEYSTORE_PASSWORD}
      ssl.key.password: ${KAFKA_KEY_PASSWORD}

Kafka Monitoring (JMX/Prometheus)

Key metrics: under-replicated partitions, offline partitions, consumer group lag, request latency, and network throughput — export via JMX exporter to Prometheus/Grafana.

✓Under-replicated partitions and offline partitions are the two most critical cluster-health metrics — both must be 0.
✓Active controller count must always equal exactly 1; 0 means no leader elected, >1 means split-brain.
✓Consumer lag = log-end-offset − committed-offset; persistent growth means consumer throughput < producer throughput.
✓JMX Exporter sidecar converts JMX MBeans to Prometheus format; Kafka Exporter adds consumer lag metrics.
✓Use kafka-consumer-groups.sh --reset-offsets to manually reset lag after a consumer bug is fixed.
✓Monitor disk usage per broker — Kafka does not back-pressure producers when brokers fill up, causing broker crashes.

Shell + YAML — JMX Exporter agent config

# Download jmx_prometheus_javaagent jar and a kafka.yaml rules file
# Start Kafka with agent:
export KAFKA_OPTS="-javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent.jar=9404:/opt/jmx_exporter/kafka.yaml"

# kafka.yaml (minimal rules for key metrics)
lowercaseOutputName: true
rules:
  # Under-replicated partitions (critical: must be 0)
  - pattern: "kafka.server<type=ReplicaManager, name=UnderReplicatedPartitions><>Value"
    name: kafka_server_replica_manager_under_replicated_partitions

  # Offline partitions (critical: must be 0)
  - pattern: "kafka.controller<type=KafkaController, name=OfflinePartitionsCount><>Value"
    name: kafka_controller_offline_partitions_count

  # Active controller (must be exactly 1 in cluster)
  - pattern: "kafka.controller<type=KafkaController, name=ActiveControllerCount><>Value"
    name: kafka_controller_active_controller_count

  # Request latency per request type
  - pattern: "kafka.network<type=RequestMetrics, name=TotalTimeMs, request=(\w+)><>Mean"
    name: kafka_network_request_total_time_ms
    labels:
      request: "$1"

Kafka Security: SSL & SASL

Kafka supports SSL/TLS for encryption-in-transit and SASL mechanisms (PLAIN, SCRAM, GSSAPI/Kerberos, OAUTHBEARER) for authentication. ACLs control which principals can produce/consume from which topics.

✓SSL/TLS encrypts data in transit; mTLS also authenticates clients
✓SASL/SCRAM-SHA-256 is the most common username/password mechanism
✓GSSAPI/Kerberos is standard in enterprise/Hadoop environments
✓Kafka ACLs use Allow/Deny rules per principal, topic, and operation
✓Super users bypass ACLs — protect super.users carefully

Kafka — SASL/SCRAM + TLS + ACLs

# Broker — server.properties
listeners=SASL_SSL://0.0.0.0:9093
security.inter.broker.protocol=SASL_SSL
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256
sasl.enabled.mechanisms=SCRAM-SHA-256
ssl.keystore.location=/certs/kafka.keystore.jks
ssl.keystore.password=changeit
ssl.truststore.location=/certs/kafka.truststore.jks
ssl.truststore.password=changeit

# Spring Boot client — application.properties
spring.kafka.security.protocol=SASL_SSL
spring.kafka.properties.sasl.mechanism=SCRAM-SHA-256
spring.kafka.properties.sasl.jaas.config=\
  org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="app-user" password="secret";

# Create ACL — allow app-user to produce to orders
kafka-acls.sh --bootstrap-server kafka:9093 \
  --add --allow-principal User:app-user \
  --producer --topic orders

Monitoring with Prometheus & Grafana

Kafka exposes hundreds of JMX metrics. Export them to Prometheus via JMX Exporter and visualise in Grafana for production-grade observability of brokers, producers, and consumers.

✓UnderReplicatedPartitions must always be 0 — any non-zero is critical
✓OfflinePartitionsCount > 0 means data is unavailable — page immediately
✓Consumer lag is the most actionable metric for application teams
✓RequestHandlerAvgIdlePercent < 30% indicates broker CPU saturation
✓Use kafka-lag-exporter for per-partition consumer lag in Prometheus

JMX Exporter + Prometheus alert

# JMX Exporter as Java agent
KAFKA_OPTS="-javaagent:/opt/jmx_prometheus_javaagent.jar=7071:/opt/kafka-jmx.yml"

# kafka-jmx.yml
rules:
  - pattern: 'kafka.server<type=ReplicaManager, name=UnderReplicatedPartitions><>Value'
    name: kafka_server_under_replicated_partitions

  - pattern: 'kafka.server<type=BrokerTopicMetrics, name=MessagesInPerSec, topic=(.+)><>OneMinuteRate'
    name: kafka_server_messages_in_per_sec
    labels:
      topic: "$1"

# Prometheus alert — consumer lag
- alert: KafkaConsumerHighLag
  expr: sum(kafka_consumer_group_lag{group="order-processor"}) > 10000
  for: 5m
  labels:
    severity: warning

Consumer Group Management

Managing Kafka consumer groups involves listing groups, checking offsets and lag, resetting offsets for replay, and deleting stale groups — essential ops skills for production incidents.

✓Offset reset requires the consumer group to be inactive
✓--to-earliest replays all retained messages; --to-latest skips to now
✓--to-datetime is most useful for "replay the last N hours"
✓Dry run first with --reset-offsets without --execute to preview
✓offsets.retention.minutes (default 7 days) — group offsets expire if inactive

Kafka CLI — consumer group operations

# List all consumer groups
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list

# Describe a group (partition owners, offsets, lag)
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group order-processor --describe

# Reset to earliest (replay all messages)
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group order-processor --topic orders \
  --reset-offsets --to-earliest --execute

# Reset to specific datetime (replay last 24 hours)
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group order-processor --topic orders \
  --reset-offsets --to-datetime 2024-01-01T00:00:00.000 --execute

# Delete a stale group (must be inactive)
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group stale-group --delete

Learn this free with Aria, your AI tutor → AiCanCode.org/learn/kafka

Cheat Sheet

Administration

Apache Kafka

8 topics · swipe →

AiCanCode.org

Administration

Log Compaction

✓cleanup.policy=compact retains the latest value per key indefinitely; this is different from time/size-based retention which deletes old segments.
✓Compaction is asynchronous — the "head" of the log always contains all recent records; only the "tail" is compacted.
✓A tombstone (key + null value) signals deletion; it is retained for delete.retention.ms before being purged.
✓Log compaction is the backbone of Kafka Streams KTable state stores — it allows state to be rebuilt by replaying the compacted changelog.
✓Kafka Connect CDC pipelines (Debezium) use compacted topics so consumers can reconstruct current DB state by replaying from offset 0.

# Create a compacted topic
kafka-topics.sh --bootstrap-server localhost:9092 \
    --create \
    --topic user-profiles \
    --partitions 6 \
    --replication-factor 3 \
    --config cleanup.policy=compact \
    --config min.insync.replicas=2 \
    --config min.compaction.lag.ms=60000  # records survive at least 60s before compaction
…

AiCanCode.org

Administration

Retention Policies

✓Retention is segment-based — old segments are deleted as a unit when the oldest message in them expires.
✓retention.ms and retention.bytes are independent; either can trigger deletion; -1 disables that dimension.
✓Log compaction retains only the latest message per key — use it for CDC and materialised-view topics.
✓A null-value message (tombstone) marks a key for deletion from a compacted topic; it is retained for delete.retention.ms before removal.
✓Segment size affects deletion granularity — a 1 GB segment means up to 1 GB of messages older than retention.ms may be kept until the segment rolls.

# Broker defaults (server.properties)
log.retention.ms=604800000      # 7 days (default)
log.retention.bytes=-1          # unlimited by size (default)
log.segment.bytes=1073741824    # 1 GB per log segment file
log.segment.ms=604800000        # roll segment after 7 days even if not full

# Create topic with custom retention
kafka-topics.sh --bootstrap-server localhost:9092   --create --topic order-events   --partitions 12 --replication-factor 3   --config retention.ms=86400000     # 24 hours
  --config retention.bytes=10737418240  # 10 GB max
…

AiCanCode.org

Administration

Topic Configuration & Tuning

✓Safety durability pattern: replication.factor=3, min.insync.replicas=2, producer acks=all — tolerates 1 broker failure without data loss
✓unclean.leader.election.enable=false prevents an out-of-sync replica from becoming leader and causing silent data loss
✓cleanup.policy=delete removes old data by time/size; cleanup.policy=compact keeps only the latest value per message key
✓max.message.bytes must be consistent across topic config, broker replica.fetch.max.bytes, and consumer max.partition.fetch.bytes
✓Smaller segment.bytes speeds up compaction and log deletion but increases open file handle count on brokers

# Create topic with durability settings
kafka-topics.sh --bootstrap-server broker:9092 \
  --create --topic payments \
  --partitions 12 \
  --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config unclean.leader.election.enable=false

# Alter existing topic config
…

AiCanCode.org

Administration

Kafka Security (TLS & SASL)

✓Kafka security = TLS (encryption) + SASL (authentication) + ACLs (authorisation) — all three work independently or together
✓SASL/PLAIN transmits credentials in plaintext — always pair with TLS (use SASL_SSL protocol, not SASL_PLAINTEXT)
✓SASL/SCRAM-SHA-256/512 stores salted credential hashes — safer than PLAIN; use for non-Kerberos environments
✓Consumer groups also need ACLs — READ permission on the consumer group is required alongside READ on the topic
✓super.users in server.properties bypass all ACL checks — restrict this list to broker inter-communication only

# server.properties — broker TLS configuration
listeners=PLAINTEXT://:9092,SSL://:9093
advertised.listeners=PLAINTEXT://broker1:9092,SSL://broker1:9093
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL

ssl.keystore.location=/etc/kafka/ssl/broker.keystore.jks
ssl.keystore.password=keystore-password
ssl.key.password=key-password
ssl.truststore.location=/etc/kafka/ssl/broker.truststore.jks
…

AiCanCode.org

Administration

Kafka Monitoring (JMX/Prometheus)

✓Under-replicated partitions and offline partitions are the two most critical cluster-health metrics — both must be 0.
✓Active controller count must always equal exactly 1; 0 means no leader elected, >1 means split-brain.
✓Consumer lag = log-end-offset − committed-offset; persistent growth means consumer throughput < producer throughput.
✓JMX Exporter sidecar converts JMX MBeans to Prometheus format; Kafka Exporter adds consumer lag metrics.
✓Use kafka-consumer-groups.sh --reset-offsets to manually reset lag after a consumer bug is fixed.

# Download jmx_prometheus_javaagent jar and a kafka.yaml rules file
# Start Kafka with agent:
export KAFKA_OPTS="-javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent.jar=9404:/opt/jmx_exporter/kafka.yaml"

# kafka.yaml (minimal rules for key metrics)
lowercaseOutputName: true
rules:
  # Under-replicated partitions (critical: must be 0)
  - pattern: "kafka.server<type=ReplicaManager, name=UnderReplicatedPartitions><>Value"
…

AiCanCode.org

Administration

Kafka Security: SSL & SASL

✓SSL/TLS encrypts data in transit; mTLS also authenticates clients
✓SASL/SCRAM-SHA-256 is the most common username/password mechanism
✓GSSAPI/Kerberos is standard in enterprise/Hadoop environments
✓Kafka ACLs use Allow/Deny rules per principal, topic, and operation
✓Super users bypass ACLs — protect super.users carefully

# Broker — server.properties
listeners=SASL_SSL://0.0.0.0:9093
security.inter.broker.protocol=SASL_SSL
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256
sasl.enabled.mechanisms=SCRAM-SHA-256
ssl.keystore.location=/certs/kafka.keystore.jks
ssl.keystore.password=changeit
ssl.truststore.location=/certs/kafka.truststore.jks
ssl.truststore.password=changeit
…

AiCanCode.org

Administration

Monitoring with Prometheus & Grafana

✓UnderReplicatedPartitions must always be 0 — any non-zero is critical
✓OfflinePartitionsCount > 0 means data is unavailable — page immediately
✓Consumer lag is the most actionable metric for application teams
✓RequestHandlerAvgIdlePercent < 30% indicates broker CPU saturation
✓Use kafka-lag-exporter for per-partition consumer lag in Prometheus

# JMX Exporter as Java agent
KAFKA_OPTS="-javaagent:/opt/jmx_prometheus_javaagent.jar=7071:/opt/kafka-jmx.yml"

# kafka-jmx.yml
rules:
  - pattern: 'kafka.server<type=ReplicaManager, name=UnderReplicatedPartitions><>Value'
    name: kafka_server_under_replicated_partitions

  - pattern: 'kafka.server<type=BrokerTopicMetrics, name=MessagesInPerSec, topic=(.+)><>OneMinuteRate'
…

AiCanCode.org

Administration

Consumer Group Management

✓Offset reset requires the consumer group to be inactive
✓--to-earliest replays all retained messages; --to-latest skips to now
✓--to-datetime is most useful for "replay the last N hours"
✓Dry run first with --reset-offsets without --execute to preview
✓offsets.retention.minutes (default 7 days) — group offsets expire if inactive

# List all consumer groups
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list

# Describe a group (partition owners, offsets, lag)
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group order-processor --describe

# Reset to earliest (replay all messages)
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
…

AiCanCode.org

Master Administration

Full lessons + an AI tutor, free.

AiCanCode.org

/learn/kafka