Unified Batch and Streaming Lakehouse for Bank Transactions

Added Jun 2025 3 design docs

Financial institutions live in two speeds at once: overnight batch files from core banking systems and real-time transaction streams from cards and payments. Keeping those worlds in separate silos produces inconsistent reporting and delayed fraud visibility, which is why the industry is converging on lakehouse architectures that process both within one governed platform. The intern builds that hybrid platform on Databricks and Spark. Daily CSV batch files and real-time Kafka transaction streams are ingested into a Medallion Architecture pipeline, with Delta Lake providing ACID guarantees, schema evolution as upstream formats drift, and time travel for auditability. A CDC flow captures changes from an operational MySQL database, Delta Live Tables apply business rules and data quality checks through the Silver layer, and Gold tables aggregate balances, volumes, and merchant analytics using SQL and Python transformations. Curated outputs are exposed to Tableau for business reporting, and Unity Catalog with role-based access control enforces who may see which financial data. The intern demonstrates the complete skill set of a modern data engineer in finance: unifying batch and streaming, database change capture, governed multi-layer modeling, and delivering BI-ready data with security controls a regulator would expect.

Related projects

Farm Sensor Data Ingestion Pipeline with Kafka and HDFS

Modern farms generate continuous streams of readings for soil moisture, temperature, humidity, and equipment status, but the raw numbers are useless until they are reliably collected, stored, and summarized. Agritech companies solve this with streaming data platforms, and this project introduces that architecture at a scale a newcomer can build end to end. The intern writes Python producers that simulate realistic agricultural sensor streams and publish them to Kafka topics, then builds consumers that read the streams, validate and enrich the readings, and land them in HDFS for durable, distributed storage. On top of the stored data, Python analysis scripts compute aggregates such as average soil moisture per field, daily temperature ranges, and gaps that indicate sensor failures, producing summary files a farm manager could act on. The focus stays on the pipeline itself: partitioning topics sensibly, handling malformed messages, and organizing HDFS directories for easy downstream analysis. Through this big-data project the intern gains a concrete understanding of publish-subscribe streaming, distributed file storage, and pipeline design, which are foundational skills for any data engineering role.

agriculturetechnologybig-datahdfs+2

View project

Medium

Energy Consumption Batch Processing and Reporting Pipeline

Utilities and large facilities collect meter readings from thousands of households and industrial sites, and the questions that matter, such as which sites are trending upward, where peak loads occur, and how weekday usage differs from weekends, require processing volumes of data that outgrow a single machine. Batch analytics on a distributed stack is the industry-standard answer. In this project the intern designs that stack from ingestion to report. Simulated household and industrial consumption records are streamed through Kafka, landed in HDFS for scalable storage, and processed in scheduled batches with PySpark jobs that compute usage patterns, peak periods, and per-segment summaries. Python scripts tie the workflow together, trigger the processing runs, and export findings as CSV reports that an operations team could consume directly. The intern must think about partitioning, job efficiency, and verifying results rather than just producing numbers. The project builds practical big-data competence in the energy domain: designing a batch pipeline, writing distributed transformations, and delivering trustworthy summary reports, all skills that transfer directly to entry-level data engineering work.

businessenergybig-datahdfs+3

View project

Medium

Model Interpretability Dashboard Using SHAP and LIME for Credit Decisions

In lending and other regulated corners of finance, a model that cannot be explained cannot be shipped: risk officers, auditors, and regulators all expect a clear account of why a credit model scored an applicant the way it did. Data science teams need tooling that makes interpretability routine rather than a one-off notebook exercise. The intern builds a Streamlit dashboard where users upload a trained scikit-learn model together with a dataset, then explore global behavior through SHAP summary and feature importance plots and drill into individual predictions with local SHAP and LIME explanations. Pandas and NumPy handle the data wrangling behind the scenes, while the interface guides users from a dataset overview to per-row explanations and makes it easy to see where the two explanation techniques agree and where they diverge — a comparison that builds real judgment about when to trust each method. The project teaches the intern how modern explainability methods work mechanically, how to build analyst-facing tooling in pure Python, and how to communicate model behavior to non-technical stakeholders — a combination finance employers explicitly screen for when hiring into model risk and credit teams.

financedata-sciencemachine-learning+6

View project