Change Data Capture

Change data capture streams committed database changes as events, letting downstream systems react without dual writes or polling every table.

data-integration

Definition

Change data capture (CDC) reads the database's log or a comparable change stream and publishes row-level inserts, updates, and deletes as ordered events. Consumers apply the changes to search indexes, caches, analytics stores, or other services. CDC observes committed changes rather than application-level dual writes, so it avoids the inconsistency of writing the database and a broker separately.

Context and scope

CDC is used for search indexing, cache invalidation, analytics ingestion, cross-service data replication, and migrating between databases. It is especially useful when the source application cannot be changed to publish events.

Common misconceptions

CDC guarantees exactly-once delivery — most streams are at-least-once; consumers must deduplicate
CDC captures uncommitted changes — it reads committed log records, and transaction boundaries matter
A snapshot is optional — a new consumer needs a consistent snapshot plus streaming from the corresponding log position

Anti-patterns

Replacing application events with raw row changes when consumers need business meaning
Ignoring schema changes, so a new column breaks downstream deserialization
Starting a consumer at the latest offset without a snapshot, silently missing existing rows

Related concepts

Authoritative source

https://debezium.io/documentation/reference/stable/architecture.html

Observe the commit log

Application-level dual writes are fragile: the database commit can succeed while the event publish fails. CDC reads the database log after commit and emits the change. The source of truth stays the database, and the stream is derived from it. A connector tracks a log position and resumes from there after a restart.

{
  "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
  "database.hostname": "db",
  "database.dbname": "app",
  "table.include.list": "public.orders",
  "plugin.name": "pgoutput",
  "slot.name": "orders_cdc"
}

The example is a Debezium connector configuration. The details vary by platform; the invariant is a durable log position and a consistent starting snapshot.

Snapshot plus stream

A new consumer needs the existing rows and every change after the snapshot. The connector performs a consistent snapshot, records the corresponding log position, and then streams changes from that position. Starting at the latest offset without a snapshot produces a downstream store that is missing all historical data. Starting a snapshot without the correct position can duplicate or skip changes.

Delivery and schema

CDC streams are usually at-least-once. Consumers must be idempotent by primary key or event id. Row changes describe the database, not necessarily the business event; a consumer that needs “order shipped” semantics should read an outbox or domain event instead of inferring intent from column updates. Schema changes require a compatible evolution strategy: add columns tolerantly, version the event envelope, and pause the pipeline for destructive changes.

When to use it

Use CDC when other systems need to react to committed database changes and the application cannot reliably publish events itself. The advantage is decoupling without dual writes; the drawback is operational complexity: replication slots, connector uptime, schema evolution, and snapshot size. Compared to an outbox, CDC is less invasive but exposes storage-level detail and depends on database-specific tooling.