Starter MySQL Change Feed Ingestion into Delta Bronze

Added Jun 2025 3 design docs

Analytics platforms need a continuous copy of operational data, and the industry-standard first step is landing database changes into a raw Bronze layer, yet most beginners have only ever loaded static CSV files. Learning to capture inserts and updates from a live relational database is the gateway skill for real data engineering work in finance and beyond. The intern builds a focused pipeline on Databricks that ingests change data from a MySQL table, for example a transactions table receiving new and updated rows, into a Delta Lake Bronze table using Spark. The project covers connecting to the source database from Python, identifying changed records, writing them to Delta format with simple schema enforcement so unexpected columns or types are caught rather than silently absorbed, and adding ingestion metadata such as load timestamps. Repeated runs demonstrate incremental behavior: only new changes land, and the Bronze table grows as an append-only history of what happened in the source. The project gives the intern their first genuine big data ingestion pattern, teaches why Bronze layers preserve raw fidelity instead of transforming eagerly, and provides hands-on Delta Lake and Spark experience, a compact foundation that the full CDC, merge, and dimensional modeling techniques build directly upon.

Related projects

Model Interpretability Dashboard Using SHAP and LIME for Credit Decisions

In lending and other regulated corners of finance, a model that cannot be explained cannot be shipped: risk officers, auditors, and regulators all expect a clear account of why a credit model scored an applicant the way it did. Data science teams need tooling that makes interpretability routine rather than a one-off notebook exercise. The intern builds a Streamlit dashboard where users upload a trained scikit-learn model together with a dataset, then explore global behavior through SHAP summary and feature importance plots and drill into individual predictions with local SHAP and LIME explanations. Pandas and NumPy handle the data wrangling behind the scenes, while the interface guides users from a dataset overview to per-row explanations and makes it easy to see where the two explanation techniques agree and where they diverge — a comparison that builds real judgment about when to trust each method. The project teaches the intern how modern explainability methods work mechanically, how to build analyst-facing tooling in pure Python, and how to communicate model behavior to non-technical stakeholders — a combination finance employers explicitly screen for when hiring into model risk and credit teams.

financedata-sciencemachine-learning+6

View project

Easy

Loan Default Risk Estimator With Random Forest and Interactive Inputs

Before extending credit, lenders need a defensible estimate of default risk — and loan officers need to explore that risk interactively, asking how the picture changes if income were higher or the loan amount smaller. This project builds a tool for exactly that kind of what-if analysis. The intern trains a random forest classifier with scikit-learn on loan applicant data, then builds a Streamlit interface where each input feature — income, loan amount, credit history, and so on — is controlled by a slider or selector. As users adjust the inputs, the app recomputes the default probability live and displays it with clear visual risk indicators, turning the model into an exploratory instrument rather than a static scorer. Feature importance charts show which factors the forest leans on most heavily, connecting the model's behavior back to intuitions about creditworthiness. The project teaches ensemble tree models on tabular financial data, probability-based classification output, and the design of interfaces that let non-technical users interrogate a model safely. The intern demonstrates they can carry a finance use case from raw dataset to an interactive decision-support tool.

financedata-sciencemachine-learning+3

View project

Medium

Ridge and Lasso Regularization Tuner for Financial Modeling

Financial datasets are full of correlated, noisy features, and ordinary regression happily overfits them. Ridge and Lasso regularization are the standard defenses, but choosing between them — and tuning their strength — requires seeing how each one reshapes a model. This project builds a tool that makes those effects directly observable. The intern develops a web tool where users upload a dataset, apply feature scaling, and train Ridge and Lasso models from scikit-learn side by side, with GridSearchCV sweeping regularization strengths under cross-validation. Coefficient visualizations show the two techniques' signature behaviors — Ridge shrinking weights smoothly toward zero while Lasso zeroes some out entirely, performing implicit feature selection — alongside evaluation metrics for each configuration. The workflow is prototyped in Jupyter with pandas and NumPy, then deployed as a FastAPI microservice so tuned models become reachable over REST. The intern gains a working command of regularization, cross-validated tuning, and the reasoning behind sparse versus dense solutions, plus deployment experience packaging an analytical workflow as a service — a combination directly relevant to quantitative finance teams.

financedata-sciencefastapi+6

View project