Cross-Source Financial Analytics with Presto and Data Governance

Added Jun 2025 3 design docs

Financial data never lives in one place: trading records sit in a warehouse, historical archives in object storage, and reference data in relational databases, and analysts lose days to moving data around before they can even ask a question. Federated query engines dissolve that friction by querying data where it lives, but in finance they must arrive with lineage and access control attached. The intern builds a federated analytics platform where Presto executes ad-hoc SQL across Hive tables, S3 data, and RDBMS sources in a single query. Ingestion flows through Kafka Connect and Flume into HDFS and S3, Spark handles transformation workloads in Python, and Apache Iceberg manages versioned table formats supporting time travel and safe schema evolution on the analytical datasets. Governance is a core deliverable, not an afterthought: Apache Atlas captures metadata and data lineage so analysts can trace any figure to its source, while Apache Ranger enforces fine-grained access policies determining who can query which tables and columns. The intern demonstrates mastery of federated architecture and the governance tooling that regulated industries require, showing they can design platforms where analytical freedom and institutional control coexist, precisely the profile financial data teams recruit for.

Related projects

Farm Sensor Data Ingestion Pipeline with Kafka and HDFS

Modern farms generate continuous streams of readings for soil moisture, temperature, humidity, and equipment status, but the raw numbers are useless until they are reliably collected, stored, and summarized. Agritech companies solve this with streaming data platforms, and this project introduces that architecture at a scale a newcomer can build end to end. The intern writes Python producers that simulate realistic agricultural sensor streams and publish them to Kafka topics, then builds consumers that read the streams, validate and enrich the readings, and land them in HDFS for durable, distributed storage. On top of the stored data, Python analysis scripts compute aggregates such as average soil moisture per field, daily temperature ranges, and gaps that indicate sensor failures, producing summary files a farm manager could act on. The focus stays on the pipeline itself: partitioning topics sensibly, handling malformed messages, and organizing HDFS directories for easy downstream analysis. Through this big-data project the intern gains a concrete understanding of publish-subscribe streaming, distributed file storage, and pipeline design, which are foundational skills for any data engineering role.

agriculturetechnologybig-datahdfs+2

View project

Medium

Energy Consumption Batch Processing and Reporting Pipeline

Utilities and large facilities collect meter readings from thousands of households and industrial sites, and the questions that matter, such as which sites are trending upward, where peak loads occur, and how weekday usage differs from weekends, require processing volumes of data that outgrow a single machine. Batch analytics on a distributed stack is the industry-standard answer. In this project the intern designs that stack from ingestion to report. Simulated household and industrial consumption records are streamed through Kafka, landed in HDFS for scalable storage, and processed in scheduled batches with PySpark jobs that compute usage patterns, peak periods, and per-segment summaries. Python scripts tie the workflow together, trigger the processing runs, and export findings as CSV reports that an operations team could consume directly. The intern must think about partitioning, job efficiency, and verifying results rather than just producing numbers. The project builds practical big-data competence in the energy domain: designing a batch pipeline, writing distributed transformations, and delivering trustworthy summary reports, all skills that transfer directly to entry-level data engineering work.

businessenergybig-datahdfs+3

View project

Medium

Model Interpretability Dashboard Using SHAP and LIME for Credit Decisions

In lending and other regulated corners of finance, a model that cannot be explained cannot be shipped: risk officers, auditors, and regulators all expect a clear account of why a credit model scored an applicant the way it did. Data science teams need tooling that makes interpretability routine rather than a one-off notebook exercise. The intern builds a Streamlit dashboard where users upload a trained scikit-learn model together with a dataset, then explore global behavior through SHAP summary and feature importance plots and drill into individual predictions with local SHAP and LIME explanations. Pandas and NumPy handle the data wrangling behind the scenes, while the interface guides users from a dataset overview to per-row explanations and makes it easy to see where the two explanation techniques agree and where they diverge — a comparison that builds real judgment about when to trust each method. The project teaches the intern how modern explainability methods work mechanically, how to build analyst-facing tooling in pure Python, and how to communicate model behavior to non-technical stakeholders — a combination finance employers explicitly screen for when hiring into model risk and credit teams.

financedata-sciencemachine-learning+6

View project