Change Data Capture Pipeline from MySQL to the Gold Layer

Added Jun 2025 3 design docs

Operational banking databases change constantly, accounts update, transactions post, customer records correct themselves, and analytics platforms that rely on nightly full copies are always stale and increasingly expensive. Change Data Capture solves this by shipping only the inserts, updates, and deletes, but applying those changes correctly downstream is where real engineering skill shows. The intern implements a CDC pipeline for financial data on Databricks. Change streams from a MySQL source are ingested into a Bronze Delta Lake layer, then applied to Silver tables using Delta Lake's MERGE INTO to upsert changes accurately. In the Gold layer the intern maintains Slowly Changing Dimensions, preserving the history of attributes like a customer's address or account status rather than overwriting it, and uses Delta Change Feed to propagate incremental updates efficiently between layers. DLT pipelines structure the transformations with data quality enforcement, and curated tables feed Databricks SQL reports for business analysis, with Spark, Python, and SQL doing the heavy lifting throughout. The project teaches merge semantics, SCD modeling, and incremental processing, the exact competencies data engineering interviews probe, and demonstrates the intern can keep an analytical platform faithfully synchronized with a live financial system.

Related projects

Model Interpretability Dashboard Using SHAP and LIME for Credit Decisions

In lending and other regulated corners of finance, a model that cannot be explained cannot be shipped: risk officers, auditors, and regulators all expect a clear account of why a credit model scored an applicant the way it did. Data science teams need tooling that makes interpretability routine rather than a one-off notebook exercise. The intern builds a Streamlit dashboard where users upload a trained scikit-learn model together with a dataset, then explore global behavior through SHAP summary and feature importance plots and drill into individual predictions with local SHAP and LIME explanations. Pandas and NumPy handle the data wrangling behind the scenes, while the interface guides users from a dataset overview to per-row explanations and makes it easy to see where the two explanation techniques agree and where they diverge — a comparison that builds real judgment about when to trust each method. The project teaches the intern how modern explainability methods work mechanically, how to build analyst-facing tooling in pure Python, and how to communicate model behavior to non-technical stakeholders — a combination finance employers explicitly screen for when hiring into model risk and credit teams.

financedata-sciencemachine-learning+6

View project

Easy

Loan Default Risk Estimator With Random Forest and Interactive Inputs

Before extending credit, lenders need a defensible estimate of default risk — and loan officers need to explore that risk interactively, asking how the picture changes if income were higher or the loan amount smaller. This project builds a tool for exactly that kind of what-if analysis. The intern trains a random forest classifier with scikit-learn on loan applicant data, then builds a Streamlit interface where each input feature — income, loan amount, credit history, and so on — is controlled by a slider or selector. As users adjust the inputs, the app recomputes the default probability live and displays it with clear visual risk indicators, turning the model into an exploratory instrument rather than a static scorer. Feature importance charts show which factors the forest leans on most heavily, connecting the model's behavior back to intuitions about creditworthiness. The project teaches ensemble tree models on tabular financial data, probability-based classification output, and the design of interfaces that let non-technical users interrogate a model safely. The intern demonstrates they can carry a finance use case from raw dataset to an interactive decision-support tool.

financedata-sciencemachine-learning+3

View project

Medium

Ridge and Lasso Regularization Tuner for Financial Modeling

Financial datasets are full of correlated, noisy features, and ordinary regression happily overfits them. Ridge and Lasso regularization are the standard defenses, but choosing between them — and tuning their strength — requires seeing how each one reshapes a model. This project builds a tool that makes those effects directly observable. The intern develops a web tool where users upload a dataset, apply feature scaling, and train Ridge and Lasso models from scikit-learn side by side, with GridSearchCV sweeping regularization strengths under cross-validation. Coefficient visualizations show the two techniques' signature behaviors — Ridge shrinking weights smoothly toward zero while Lasso zeroes some out entirely, performing implicit feature selection — alongside evaluation metrics for each configuration. The workflow is prototyped in Jupyter with pandas and NumPy, then deployed as a FastAPI microservice so tuned models become reachable over REST. The intern gains a working command of regularization, cross-validated tuning, and the reasoning behind sparse versus dense solutions, plus deployment experience packaging an analytical workflow as a service — a combination directly relevant to quantitative finance teams.

financedata-sciencefastapi+6

View project