Home/Learn/Apache Kafka/MirrorMaker 2 for Geo-Replication

MirrorMaker 2 for Geo-Replication

Advanced

MirrorMaker 2 (MM2) is a Kafka Connect-based tool that replicates topics across clusters for disaster recovery, data locality, or multi-region active-active deployments.

Overview

MirrorMaker 2 is built on Kafka Connect and uses the MirrorSourceConnector, MirrorCheckpointConnector, and MirrorHeartbeatConnector to replicate topics, consumer group offsets, and heartbeats between clusters. It preserves relative consumer offsets across clusters, enabling seamless failover. MM2 prefixes replicated topics with the source alias (e.g. us-east.orders) to avoid naming collisions. It supports active-passive DR, active-active multi-region, and data locality for regulations like GDPR.

MM2 Architecture & Connectors

MirrorSourceConnector replicates topic data. MirrorCheckpointConnector translates source consumer group offsets to target cluster offsets and emits checkpoints. MirrorHeartbeatConnector writes periodic heartbeats to measure replication lag.

Properties — MirrorMaker 2 configuration

# mm2.properties — standalone or Connect distributed worker config
clusters = us-east, eu-west

us-east.bootstrap.servers = kafka-us-east:9092
eu-west.bootstrap.servers = kafka-eu-west:9092

# Replicate from us-east to eu-west
us-east->eu-west.enabled = true
us-east->eu-west.topics = orders, payments, inventory.*

# Offset and heartbeat replication
us-east->eu-west.emit.checkpoints.enabled = true
us-east->eu-west.emit.heartbeats.enabled  = true
us-east->eu-west.sync.group.offsets.enabled = true

# Replication factor on the target cluster
replication.factor = 3

# Topic renaming — "orders" on us-east becomes "us-east.orders" on eu-west
# (override with replication.policy.class for custom naming)

Consumer Failover with Translated Offsets

MirrorCheckpointConnector stores translated offsets in an internal topic. On failover, consumers use the RemoteClusterUtils helper or set auto.offset.reset=latest on the target cluster. The translated offset lets them resume near where they left off.

Java — offset translation for failover

// Translate consumer group offsets before failover
Map<TopicPartition, OffsetAndMetadata> translatedOffsets =
    RemoteClusterUtils.translateOffsets(
        props,
        "us-east",                 // source cluster alias
        "order-processor",         // consumer group
        Duration.ofSeconds(10)
    );

// Seek the target consumer to translated offsets
KafkaConsumer<String, OrderEvent> consumer = new KafkaConsumer<>(targetProps);
consumer.assign(translatedOffsets.keySet());
translatedOffsets.forEach(consumer::seek);

// Then poll normally from eu-west cluster
while (true) {
    ConsumerRecords<String, OrderEvent> records =
        consumer.poll(Duration.ofMillis(200));
    // process ...
}

Active-Active Deployment & Cycle Prevention

In active-active setups both clusters accept writes and replicate to each other. MM2 uses DefaultReplicationPolicy to avoid replication cycles: a topic already prefixed with a remote alias is not re-replicated back.

Properties + Shell — active-active config and monitoring

# Active-Active: us-east ↔ eu-west
us-east->eu-west.enabled = true
eu-west->us-east.enabled = true

# Topics to replicate (both directions)
us-east->eu-west.topics = orders, payments
eu-west->us-east.topics = orders, payments

# Cycle prevention is automatic:
# "us-east.orders" (already prefixed) is NOT replicated back to us-east

# Check replication lag via heartbeats
kafka-consumer-groups.sh \
  --bootstrap-server kafka-eu-west:9092 \
  --group mirrormaker2-us-east \
  --describe

# Or use the heartbeat topic
kafka-console-consumer.sh \
  --bootstrap-server kafka-eu-west:9092 \
  --topic us-east.heartbeats \
  --from-beginning

Key Points to Remember

1MM2 is built on Kafka Connect — run it as a Connect worker with mirror connector configs.
2MirrorSourceConnector replicates data; MirrorCheckpointConnector translates offsets; MirrorHeartbeatConnector measures lag.
3Replicated topics are prefixed with source alias (us-east.orders) to prevent cycles.
4Translated offsets enable consumers to resume near their original position after failover.
5Active-active requires both directions enabled; cycle prevention is automatic via prefix policy.
6sync.group.offsets.enabled keeps consumer group positions in sync for near-seamless failover.

Interview Questions

What is MirrorMaker 2 and how does it differ from MirrorMaker 1?

MediumLinkedIn

How does MM2 prevent replication cycles in active-active setups?

MediumConfluent

Explain the role of MirrorCheckpointConnector in consumer failover.

HardUber

What is the naming convention MM2 applies to replicated topics and why?

MediumAmazon

How would you measure geo-replication lag with MirrorMaker 2?

HardNetflix

Ask Aria about MirrorMaker 2 for Geo-Replication

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

Kafka Cluster Scaling

Kafka Performance Tuning