Consumer Lag
IntermediateLag is the difference between the latest produced offset and the last committed consumer offset; high lag indicates the consumer cannot keep up with the producer rate.
Overview
Consumer lag is the gap between how far a producer has written (log-end offset) and how far a consumer has read (current offset). It is the most important operational metric for Kafka consumers — high lag means the consumer is falling behind the producer and may miss real-time SLAs. Each partition has its own lag value; total group lag is the sum across all partitions. Zero lag does not mean everything is fine — it could mean the consumer keeps up but processes slowly. Monitoring lag, its trend (growing vs stable), and the time-to-catch-up gives a complete picture of consumer health.
Measuring Lag — CLI & Admin API
Kafka's kafka-consumer-groups.sh tool shows per-partition lag instantly. For programmatic access, use the AdminClient API to call listConsumerGroupOffsets and compare to endOffsets. This is how monitoring tools like Prometheus Kafka Exporter work.
# CLI — describe a consumer group to see lag per partition
kafka-consumer-groups.sh \
--bootstrap-server localhost:9092 \
--describe \
--group order-processing-service
# Output:
# GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
# order-processing-service orders 0 5000 5000 0 ← healthy
# order-processing-service orders 1 4800 5200 400 ← lagging!
# order-processing-service orders 2 5100 5100 0
// Programmatic lag calculation
AdminClient admin = AdminClient.create(Map.of(
AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"
));
// Get committed offsets for the group
Map<TopicPartition, OffsetAndMetadata> committed =
admin.listConsumerGroupOffsets("order-processing-service")
.partitionsToOffsetAndMetadata().get();
// Get end offsets (latest produced)
Map<TopicPartition, Long> endOffsets =
admin.listOffsets(
committed.keySet().stream()
.collect(Collectors.toMap(tp -> tp, tp -> OffsetSpec.latest())))
.all().get()
.entrySet().stream()
.collect(Collectors.toMap(Map.Entry::getKey, e -> e.getValue().offset()));
// Calculate lag per partition
committed.forEach((tp, meta) -> {
long lag = endOffsets.get(tp) - meta.offset();
System.out.printf("Partition %s: lag = %d%n", tp, lag);
});Common Causes of High Lag
Understanding why lag grows points to the right fix:
**Too few consumers / partitions** — if the consumer group has fewer consumers than partitions, some consumers handle multiple partitions. Add consumers (up to the partition count) or increase partition count.
**Slow processing logic** — each record takes too long (DB calls, external APIs). Use async processing, parallel execution, or batch processing.
**max.poll.records too high** — consumer fetches a large batch but takes longer than max.poll.interval.ms to process it, causing the coordinator to kick the consumer out as dead → rebalance → lag spikes.
**Downstream bottleneck** — the consumer itself is fast but the DB/service it calls is slow. Scale the downstream system.
# Tuning to reduce lag
# 1. Increase consumer instances (up to partition count)
# In Kubernetes: scale the deployment
kubectl scale deployment order-consumer --replicas=12 # match partition count
# 2. Reduce max.poll.records to avoid poll-interval breach
spring:
kafka:
consumer:
max-poll-records: 50 # default 500 — lower if processing is slow
properties:
max.poll.interval.ms: 30000 # must exceed max single-batch processing time
# 3. Use concurrent listeners for parallel partition processing
@Bean
public ConcurrentKafkaListenerContainerFactory<String, Order> factory(
ConsumerFactory<String, Order> cf) {
var factory = new ConcurrentKafkaListenerContainerFactory<String, Order>();
factory.setConsumerFactory(cf);
factory.setConcurrency(6); // 6 consumer threads, one per partition subset
return factory;
}Alerting on Lag — Prometheus & Grafana
In production, scrape consumer lag metrics with the Kafka Exporter or JMX Exporter and alert when lag exceeds a threshold relative to your SLA. Alerting on lag velocity (growing lag) is more actionable than a simple threshold — a stable lag of 1 000 is fine; a lag growing by 100/minute is a pending incident.
# Prometheus alert rule — alert when lag > 10,000 for 5 minutes
groups:
- name: kafka
rules:
- alert: KafkaConsumerHighLag
expr: kafka_consumer_group_lag > 10000
for: 5m
labels:
severity: warning
annotations:
summary: "High consumer lag on {{ $labels.group }}/{{ $labels.topic }}/{{ $labels.partition }}"
description: "Lag is {{ $value }} — consumer may not meet SLA"
- alert: KafkaConsumerLagGrowing
expr: rate(kafka_consumer_group_lag[5m]) > 100
for: 3m
labels:
severity: critical
annotations:
summary: "Consumer lag growing rapidly for {{ $labels.group }}"Key Points to Remember
- 1Consumer lag = log-end-offset − current-committed-offset per partition; monitor per-partition, not just total.
- 2A steadily growing lag is more alarming than a high-but-stable lag — alert on lag velocity, not just threshold.
- 3Root causes: too few consumers, slow processing, max.poll.records too high causing poll-interval breach, downstream bottleneck.
- 4Max consumers in a group = partition count; adding more consumers beyond that has no effect — they sit idle.
- 5Use kafka-consumer-groups.sh or AdminClient.listConsumerGroupOffsets() to measure lag programmatically.
- 6Prometheus Kafka Exporter / JMX Exporter exposes kafka_consumer_group_lag metric for Grafana dashboards and alerting.
Interview Questions
Sign in to ask AriaHow is consumer lag calculated in Kafka?
A Kafka consumer group is lagging on partition 3 but not others. What would you investigate?
You have 12 partitions and 8 consumers. Lag is growing. What would you do?
What happens when a consumer exceeds max.poll.interval.ms and how does it affect lag?
How would you design a lag alert that avoids false positives during planned batch backfills?
Ask Aria about Consumer Lag
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.