How Change Data Capture Works
AdvancedChange Data Capture (CDC) streams every insert, update, and delete from a database to other systems in near real time. Instead of polling for changes, log-based CDC reads the database transaction log (Postgres WAL, MySQL binlog) — the same log used for replication — and emits a change event per row change. Tools like Debezium publish these to Kafka, so caches, search indexes, data warehouses, and microservices stay in sync without the database even knowing they exist.
Think of a security camera on the database
Polling for changes is like walking into a room every few minutes and comparing it to a photo to spot what moved — slow, and you miss anything that changed and changed back. Log-based CDC is a security camera recording every movement as it happens (the transaction log). You get a precise, ordered feed of exactly what changed and when, without disturbing the people in the room (no load on the primary tables) — and you can send that feed anywhere.
Step by Step
Key Concepts
Log-based vs Query-based CDC
Log-based CDC reads the transaction log for a low-overhead, complete, ordered change stream (including deletes). Query-based CDC polls for rows changed since a timestamp — simpler but misses deletes and intermediate states and adds query load.
Transaction Log (WAL/binlog)
The database durable, ordered record of every change, used for crash recovery and replication. CDC reuses it as the source of truth for change events.
Debezium
A popular open-source CDC platform that connects to databases, reads their logs, and streams change events to Kafka, so downstream systems can consume them in real time.
Transactional Outbox
A pattern where a service writes an event to an outbox table within the same transaction as its data change; CDC then publishes the outbox, guaranteeing the change and its event are atomic and consistent.
Key Facts
- Log-based CDC captures every change (including deletes) in order with minimal load, unlike polling which misses deletes and short-lived intermediate values.
- CDC plus the outbox pattern is the standard reliable answer to the dual-write problem in microservices — no lost or phantom events.
- It decouples systems: the source database is unaware of its consumers, so you can add a new downstream (a data warehouse) without touching the source app.
Real-World Applications
Keeping a search index in sync
Every change to the products table streams via CDC to update an Elasticsearch index in near real time, so search results reflect the database within seconds — without the app writing to both stores.
Reliable event publishing (outbox)
An order service writes the order and an order-created event to an outbox table in one transaction; CDC streams the outbox to Kafka, guaranteeing the event is published exactly when the order commits.
Frequently Asked Questions
What is Change Data Capture (CDC)?
CDC is a technique for streaming every insert, update, and delete from a database to other systems in near real time. Rather than periodically polling for changes, log-based CDC reads the database transaction log (the same log used for replication) and emits an event for each row change, so downstream systems like caches, search indexes, and data warehouses stay in sync automatically.
What is the difference between log-based and query-based CDC?
Log-based CDC reads the database transaction log, capturing every change in order — including deletes and intermediate values — with very little load on the main tables. Query-based CDC repeatedly queries for rows changed since a timestamp; it is simpler but adds query load, cannot easily detect deletes, and may miss values that changed and changed back between polls. Log-based is generally preferred for completeness and performance.
What problem does the transactional outbox pattern solve?
It solves the dual-write problem: when a service must both update its database and publish an event, doing them as two separate operations risks one succeeding and the other failing, leaving systems inconsistent. With the outbox pattern, the service writes the event to an outbox table in the same database transaction as the data change, so both commit atomically. CDC then streams the outbox, guaranteeing the event is published exactly when the change is durable.
What is Debezium used for?
Debezium is an open-source CDC platform that connects to databases like PostgreSQL, MySQL, and MongoDB, reads their transaction logs, and streams the resulting change events to Kafka. It lets you build real-time data pipelines — syncing caches and search indexes, feeding data warehouses, or implementing the outbox pattern — without the source database or application needing to know about the consumers.