Change Data Capture Pipeline from MySQL to the Gold Layer

Operational banking databases change constantly, accounts update, transactions post, customer records correct themselves, and analytics platforms that rely on nightly full copies are always stale and increasingly expensive. Change Data Capture solves this by shipping only the inserts, updates, and deletes, but applying those changes correctly downstream is where real engineering skill shows.

The intern implements a CDC pipeline for financial data on Databricks. Change streams from a MySQL source are ingested into a Bronze Delta Lake layer, then applied to Silver tables using Delta Lake's MERGE INTO to upsert changes accurately. In the Gold layer the intern maintains Slowly Changing Dimensions, preserving the history of attributes like a customer's address or account status rather than overwriting it, and uses Delta Change Feed to propagate incremental updates efficiently between layers. DLT pipelines structure the transformations with data quality enforcement, and curated tables feed Databricks SQL reports for business analysis, with Spark, Python, and SQL doing the heavy lifting throughout.

The project teaches merge semantics, SCD modeling, and incremental processing, the exact competencies data engineering interviews probe, and demonstrates the intern can keep an analytical platform faithfully synchronized with a live financial system.