Source Connectors & Debezium CDC
IntermediateSource connectors pull data into Kafka from external systems; the Debezium CDC connector captures database changes as Kafka events in real time using DB transaction logs.
Overview
Source connectors ingest data into Kafka from external systems. Two dominant approaches: polling-based (JDBC Source connector queries a table on a schedule, looking for new/updated rows via a timestamp or auto-increment column) and event-driven CDC (Change Data Capture). Debezium is the leading CDC connector: it reads the database binary log (MySQL binlog, Postgres WAL, MongoDB oplog) directly, capturing every INSERT, UPDATE, and DELETE as a Kafka event in real time with zero latency and zero DB query overhead. Debezium events carry the full before/after row state, making them ideal for event sourcing, cache invalidation, search index synchronisation, and microservice data replication without polling.
Debezium MySQL CDC connector
Debezium reads the MySQL binlog. Configure binlog_format=ROW on MySQL, create a replication user, then deploy the connector. Events appear on dbserver1.orders.orders topic.
-- MySQL prerequisites
-- 1. Enable binlog in my.cnf
-- [mysqld]
-- server-id=1
-- log_bin=/var/log/mysql/mysql-bin.log
-- binlog_format=ROW
-- binlog_row_image=FULL
-- 2. Create replication user
CREATE USER 'debezium'@'%' IDENTIFIED BY 'dbz_password';
GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE,
REPLICATION CLIENT ON *.* TO 'debezium'@'%';
FLUSH PRIVILEGES;
# Deploy Debezium MySQL connector
curl -X POST http://connect:8083/connectors \
-H "Content-Type: application/json" \
-d '{
"name": "mysql-cdc",
"config": {
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"database.hostname": "mysql",
"database.port": "3306",
"database.user": "debezium",
"database.password": "dbz_password",
"database.server.id": "184054",
"topic.prefix": "dbserver1",
"database.include.list": "orders",
"schema.history.internal.kafka.bootstrap.servers": "kafka:9092",
"schema.history.internal.kafka.topic": "dbhistory.orders",
"tasks.max": "1"
}
}'Debezium event structure (before/after)
Each Debezium event contains the full before and after row state, plus operation type (c=create, u=update, d=delete, r=read/snapshot).
// Debezium event on topic dbserver1.orders.orders
{
"schema": { ... },
"payload": {
"before": {
"id": 42,
"status": "PENDING",
"total": 99.99
},
"after": {
"id": 42,
"status": "SHIPPED", // updated field
"total": 99.99
},
"source": {
"version": "2.5.0",
"db": "orders",
"table": "orders",
"ts_ms": 1711123456789,
"gtid": "3E11FA47-71CA-11E1-9E33-C80AA9429562:23"
},
"op": "u", // c=insert, u=update, d=delete, r=snapshot read
"ts_ms": 1711123456900
}
}
// DELETE event: before=full row, after=null
// INSERT event: before=null, after=full row
// Consuming with Spring Kafka
@KafkaListener(topics = "dbserver1.orders.orders")
public void handleOrderChange(String payload) {
JsonNode event = objectMapper.readTree(payload);
String op = event.at("/payload/op").asText();
if ("u".equals(op)) {
String newStatus = event.at("/payload/after/status").asText();
searchIndexService.updateOrderStatus(
event.at("/payload/after/id").asLong(), newStatus);
}
}JDBC polling source vs Debezium CDC — when to use each
JDBC Source is simpler to set up; Debezium is more powerful but requires binlog access. Choose based on your requirements.
# JDBC Source connector — polling mode
# Requires: updated_at timestamp column OR auto-increment id
# Pros: simple, no special DB permissions, works with any JDBC DB
# Cons: polling interval latency (seconds), misses hard-deletes, DB load from queries
{
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"mode": "timestamp+incrementing",
"timestamp.column.name": "updated_at",
"incrementing.column.name": "id",
"poll.interval.ms": "5000"
}
# Debezium CDC
# Pros: zero latency, captures deletes, no updated_at column required,
# full before/after context, consistent snapshot on startup
# Cons: requires binlog access, replication user, schema history topic,
# exactly-one-task per connector (no horizontal scaling within connector)
# Decision guide:
# Use JDBC Source when:
# - Simple append-only tables (e.g. events, audit logs)
# - Already have updated_at columns
# - No binlog access or DBA approval for replication user
# Use Debezium when:
# - Need to capture DELETEs
# - Need real-time low-latency streaming (<1s)
# - Outbox pattern: publish events via DB INSERT, Debezium delivers to Kafka
# - Cache invalidation on every row changeKey Points to Remember
- 1Debezium reads the binary log directly — zero polling overhead, sub-second latency, captures DELETEs.
- 2JDBC Source polling requires an updated_at or auto-increment column and cannot capture hard DELETEs.
- 3Debezium events carry full before/after row state — ideal for audit, cache invalidation, and search index sync.
- 4The Outbox Pattern uses Debezium to reliably publish domain events: write to outbox table → Debezium delivers to Kafka.
- 5Debezium captures an initial snapshot of existing data before streaming new changes from the binlog.
- 6Each Debezium MySQL connector is single-task (tasks.max=1); scale by running more connectors per table subset.
Interview Questions
Sign in to ask AriaWhat is CDC and how does Debezium capture changes from a MySQL database?
Why does Debezium require binlog_format=ROW on MySQL?
How does the Outbox Pattern use Debezium to guarantee exactly-once event publishing?
What are the tradeoffs between JDBC Source polling and Debezium CDC?
How does Debezium handle the initial snapshot of existing data before streaming changes?
Ask Aria about Source Connectors & Debezium CDC
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.