Starter MySQL Change Feed Ingestion into Delta Bronze

Analytics platforms need a continuous copy of operational data, and the industry-standard first step is landing database changes into a raw Bronze layer, yet most beginners have only ever loaded static CSV files. Learning to capture inserts and updates from a live relational database is the gateway skill for real data engineering work in finance and beyond.

The intern builds a focused pipeline on Databricks that ingests change data from a MySQL table, for example a transactions table receiving new and updated rows, into a Delta Lake Bronze table using Spark. The project covers connecting to the source database from Python, identifying changed records, writing them to Delta format with simple schema enforcement so unexpected columns or types are caught rather than silently absorbed, and adding ingestion metadata such as load timestamps. Repeated runs demonstrate incremental behavior: only new changes land, and the Bronze table grows as an append-only history of what happened in the source.

The project gives the intern their first genuine big data ingestion pattern, teaches why Bronze layers preserve raw fidelity instead of transforming eagerly, and provides hands-on Delta Lake and Spark experience, a compact foundation that the full CDC, merge, and dimensional modeling techniques build directly upon.