Kafka Monitoring (JMX/Prometheus)
IntermediateKey metrics: under-replicated partitions, offline partitions, consumer group lag, request latency, and network throughput — export via JMX exporter to Prometheus/Grafana.
Overview
Kafka exposes hundreds of metrics via JMX; the prometheus-jmx-exporter sidecar converts them to Prometheus format. The most critical cluster-health metrics are under-replicated partitions (should always be 0), offline partitions (must always be 0), and active controller count (must always be 1). The most critical application metric is consumer group lag — the difference between the producer's latest offset and the consumer's committed offset. Lag > 0 means the consumer is behind; unbounded lag growth means the consumer will never catch up. Lag is exposed via the Kafka AdminClient API and the kafka-consumer-groups.sh tool, and visualised in Grafana using the Kafka Lag Exporter or Confluent's kafka-consumer-group-lag metric.
JMX Exporter sidecar setup
Add the prometheus jmx_exporter Java agent to each Kafka broker JVM. It translates JMX MBeans to Prometheus /metrics endpoint on port 9404.
# Download jmx_prometheus_javaagent jar and a kafka.yaml rules file
# Start Kafka with agent:
export KAFKA_OPTS="-javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent.jar=9404:/opt/jmx_exporter/kafka.yaml"
# kafka.yaml (minimal rules for key metrics)
lowercaseOutputName: true
rules:
# Under-replicated partitions (critical: must be 0)
- pattern: "kafka.server<type=ReplicaManager, name=UnderReplicatedPartitions><>Value"
name: kafka_server_replica_manager_under_replicated_partitions
# Offline partitions (critical: must be 0)
- pattern: "kafka.controller<type=KafkaController, name=OfflinePartitionsCount><>Value"
name: kafka_controller_offline_partitions_count
# Active controller (must be exactly 1 in cluster)
- pattern: "kafka.controller<type=KafkaController, name=ActiveControllerCount><>Value"
name: kafka_controller_active_controller_count
# Request latency per request type
- pattern: "kafka.network<type=RequestMetrics, name=TotalTimeMs, request=(\w+)><>Mean"
name: kafka_network_request_total_time_ms
labels:
request: "$1"Consumer group lag monitoring
Consumer lag is the most actionable metric. Monitor it via kafka-consumer-groups.sh, Kafka Exporter (open-source), or directly via AdminClient.
# CLI: show lag per partition
kafka-consumer-groups.sh \
--bootstrap-server localhost:9092 \
--describe --group order-processor
# Output columns:
# TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID
# Lag = LOG-END-OFFSET - CURRENT-OFFSET
# Kafka Exporter (https://github.com/danielqsj/kafka_exporter)
# Exposes:
# kafka_consumergroup_lag{consumergroup="order-processor", topic="order-events", partition="0"}
# kafka_consumergroup_current_offset
# kafka_topic_partition_current_offset
# Prometheus alert — consumer is falling behind
# - alert: KafkaConsumerLagHigh
# expr: kafka_consumergroup_lag > 10000
# for: 5m
# labels:
# severity: warning
# Spring Boot: check lag programmatically via AdminClient
AdminClient admin = AdminClient.create(Map.of(
ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"));
Map<TopicPartition, OffsetAndMetadata> committed =
admin.listConsumerGroupOffsets("order-processor")
.partitionsToOffsetAndMetadata().get();
// compare with listOffsets() to compute per-partition lagKey Prometheus alert rules for Kafka
These rules cover the most important Kafka failure modes that require immediate action.
groups:
- name: kafka
rules:
# Cluster health — data loss risk
- alert: KafkaUnderReplicatedPartitions
expr: kafka_server_replica_manager_under_replicated_partitions > 0
for: 1m
labels:
severity: critical
annotations:
summary: "{{ $value }} under-replicated partitions — data loss risk"
- alert: KafkaOfflinePartitions
expr: kafka_controller_offline_partitions_count > 0
for: 0m
labels:
severity: critical
annotations:
summary: "{{ $value }} offline partitions — data unavailable!"
# Active controller — split-brain risk
- alert: KafkaNoActiveController
expr: sum(kafka_controller_active_controller_count) != 1
for: 1m
labels:
severity: critical
# Consumer lag growth
- alert: KafkaConsumerLagGrowing
expr: |
increase(kafka_consumergroup_lag[10m]) > 1000
for: 5m
labels:
severity: warning
annotations:
summary: "Consumer group {{ $labels.consumergroup }} lag growing"Key Points to Remember
- 1Under-replicated partitions and offline partitions are the two most critical cluster-health metrics — both must be 0.
- 2Active controller count must always equal exactly 1; 0 means no leader elected, >1 means split-brain.
- 3Consumer lag = log-end-offset − committed-offset; persistent growth means consumer throughput < producer throughput.
- 4JMX Exporter sidecar converts JMX MBeans to Prometheus format; Kafka Exporter adds consumer lag metrics.
- 5Use kafka-consumer-groups.sh --reset-offsets to manually reset lag after a consumer bug is fixed.
- 6Monitor disk usage per broker — Kafka does not back-pressure producers when brokers fill up, causing broker crashes.
Interview Questions
Sign in to ask AriaWhat does consumer group lag represent and what causes it to grow unboundedly?
What should you check first when you see under-replicated partitions in a production cluster?
How would you detect and alert on a broker that has fallen out of the ISR?
What metrics would you put on a Kafka health dashboard for an on-call engineer?
How do you programmatically compute consumer lag using the Kafka AdminClient API?
Ask Aria about Kafka Monitoring (JMX/Prometheus)
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.