Home/Learn/Apache Kafka/Replication & In-Sync Replicas (ISR)

Replication & In-Sync Replicas (ISR)

Intermediate

Fundamentals

Each partition has a configurable replication factor; the ISR list tracks replicas that are fully caught up — only ISR replicas can be elected leader on failover.

Overview

Kafka guarantees fault tolerance through **partition replication**. Each partition has one leader and `replication.factor - 1` follower replicas spread across different broker nodes. Followers continuously fetch from the leader to stay current. The **In-Sync Replicas (ISR)** list is a dynamic set maintained by the controller: a replica stays in the ISR as long as it fetches from the leader within `replica.lag.time.max.ms` (default 30 s). If a follower falls behind, it is removed from the ISR. On leader failure, the controller elects a new leader **only from the current ISR** (unless `unclean.leader.election.enable = true`, which risks data loss). The producer's `acks` setting determines how many ISR replicas must confirm a write before it is acknowledged.

Replication Factor and ISR Mechanics

A `replication.factor = 3` means each partition has 3 copies on 3 different brokers. The leader handles all reads and writes. Followers pull messages in batches. The ISR is per-partition and is persisted in ZooKeeper/KRaft metadata. A replica is in-sync if its log end offset (LEO) matches the leader's within the lag threshold.

Kafka — replication factor and ISR inspection

# Create a topic with replication factor 3
kafka-topics.sh --create   --topic orders   --partitions 6   --replication-factor 3   --bootstrap-server broker1:9092

# Describe topic — see leaders, replicas, ISR per partition
kafka-topics.sh --describe --topic orders --bootstrap-server broker1:9092

# Output:
# Topic: orders  Partition: 0  Leader: 1  Replicas: 1,2,3  Isr: 1,2,3
# Topic: orders  Partition: 1  Leader: 2  Replicas: 2,3,1  Isr: 2,3,1
# (If broker 3 lags, partition 0 ISR becomes: 1,2)

# Under-replicated partitions alert — ISR < replication factor
kafka-topics.sh --describe --under-replicated-partitions   --bootstrap-server broker1:9092

acks and min.insync.replicas — Durability Knobs

`acks=0` (no wait), `acks=1` (leader only), `acks=all` (all ISR replicas). `min.insync.replicas` (broker/topic config) sets the minimum ISR size that must acknowledge for `acks=all`. With `replication.factor=3` and `min.insync.replicas=2`: the write succeeds as long as 2 replicas are in sync; if only 1 replica remains the producer gets a `NotEnoughReplicasException`.

Kafka — acks and min.insync.replicas config

# Recommended production settings
# Producer (acks=all + retries for durability)
acks=all
retries=Integer.MAX_VALUE
enable.idempotence=true

# Broker / topic
replication.factor=3
min.insync.replicas=2     # tolerate 1 broker down; fail-fast if 2 go down

# Trade-off table:
# acks=1, min.isr=1  → highest throughput, may lose messages on leader crash
# acks=all, min.isr=1 → waits for ISR but ISR might be just leader
# acks=all, min.isr=2 → strong durability, tolerates 1 failure with rf=3
# acks=all, min.isr=3 → no message loss, but any one broker down → unavailable

Unclean Leader Election — Availability vs Durability

`unclean.leader.election.enable` (default `false`) controls whether an out-of-sync replica can be elected leader when all ISR replicas are unavailable. Setting it `true` keeps the partition available but risks data loss (the elected replica may be behind). The `kafka:under_replicated_partitions` Prometheus metric is the key alert for ISR shrinkage.

Kafka — unclean leader election and monitoring

# Production: keep unclean.leader.election disabled (default false)
# Enable only for topics where availability > durability (e.g. metrics)
kafka-configs.sh --alter   --entity-type topics   --entity-name telemetry-events   --add-config unclean.leader.election.enable=true   --bootstrap-server broker1:9092

# Key Prometheus alerts
- alert: KafkaUnderReplicatedPartitions
  expr: kafka_server_replicamanager_underreplicatedpartitions > 0
  for: 5m

- alert: KafkaOfflinePartitions
  expr: kafka_controller_kafkacontroller_offlinepartitionscount > 0
  for: 1m
  annotations:
    severity: critical

Key Points to Remember

1replication.factor copies each partition to N brokers for fault tolerance
2ISR = set of replicas caught up within replica.lag.time.max.ms of the leader
3Leader election on failure picks only from the current ISR (unless unclean election enabled)
4acks=all ensures all ISR replicas acknowledge; combine with min.insync.replicas=2 for RF=3
5unclean.leader.election.enable=true risks data loss but prevents partition unavailability
6Monitor under-replicated-partitions metric — it fires before data loss occurs

Interview Questions

What is the ISR in Kafka and when is a replica removed from it?

MediumConfluent

What is the effect of setting min.insync.replicas=2 with replication.factor=3?

MediumLinkedIn

What is unclean leader election and when would you enable it?

HardUber

Why does acks=all not guarantee no data loss without min.insync.replicas?

HardConfluent

What metric would you alert on to detect ISR shrinkage before data loss?

MediumAmazon

Ask Aria about Replication & In-Sync Replicas (ISR)

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

ZooKeeper & KRaft Mode

Leaders & Followers