CDC (Change Data Capture)
Definition
Change Data Capture (CDC) is a set of software design patterns used to determine and track the data that has changed so that action can be taken using the changed data. It captures row-level changes (inserts, updates, deletes) in a database and makes them available in a stream of events.
Why It Matters
CDC is a powerful technique for data integration and building real-time, event-driven systems. It allows different systems to stay in sync with a database without having to constantly query it for changes, which is inefficient.
Contextual Example
A company uses CDC on its main production database. When a new order is created, the CDC process captures this "insert" event and publishes it to a message queue. A separate analytics system can then consume this event in real-time to update a dashboard.
Common Misunderstandings
- CDC is more efficient than batch-based ETL processes for near real-time data replication.
- It is often implemented by reading the database's transaction log (or write-ahead log).