Hybrid Batch and Streaming Analytics for E-Commerce Events

Added Jun 2025 3 design docs

E-commerce businesses ask two kinds of questions of the same data: what is happening on the site right now, and what patterns emerge over months of orders. Answering both requires a platform that marries streaming clickstream processing with heavyweight batch analytics over transactional history, coordinated so the two views stay consistent. The intern develops this hybrid platform on the open-source big data stack. Sqoop performs batch ingestion of transaction data from relational databases while Kafka carries real-time clickstream events; Spark handles both modes, with Spark Streaming computing live metrics like active sessions and conversion funnels and batch jobs building historical aggregates. HDFS provides durable storage, Hive exposes warehouse tables over the curated data, and Presto serves fast interactive queries for analysts. Airflow orchestrates the ETL workflows end to end with scheduling and dependency management, and the intern instruments the whole system with Prometheus metrics and Grafana dashboards tracking both cluster health and pipeline throughput. Completing the platform proves the intern can reason about lambda-style architectures, operate the classic Hadoop-ecosystem toolset that large retailers still run at scale, and treat monitoring as part of the deliverable rather than an afterthought.

Related projects

Farm Sensor Data Ingestion Pipeline with Kafka and HDFS

Modern farms generate continuous streams of readings for soil moisture, temperature, humidity, and equipment status, but the raw numbers are useless until they are reliably collected, stored, and summarized. Agritech companies solve this with streaming data platforms, and this project introduces that architecture at a scale a newcomer can build end to end. The intern writes Python producers that simulate realistic agricultural sensor streams and publish them to Kafka topics, then builds consumers that read the streams, validate and enrich the readings, and land them in HDFS for durable, distributed storage. On top of the stored data, Python analysis scripts compute aggregates such as average soil moisture per field, daily temperature ranges, and gaps that indicate sensor failures, producing summary files a farm manager could act on. The focus stays on the pipeline itself: partitioning topics sensibly, handling malformed messages, and organizing HDFS directories for easy downstream analysis. Through this big-data project the intern gains a concrete understanding of publish-subscribe streaming, distributed file storage, and pipeline design, which are foundational skills for any data engineering role.

agriculturetechnologybig-datahdfs+2

View project

Medium

Energy Consumption Batch Processing and Reporting Pipeline

Utilities and large facilities collect meter readings from thousands of households and industrial sites, and the questions that matter, such as which sites are trending upward, where peak loads occur, and how weekday usage differs from weekends, require processing volumes of data that outgrow a single machine. Batch analytics on a distributed stack is the industry-standard answer. In this project the intern designs that stack from ingestion to report. Simulated household and industrial consumption records are streamed through Kafka, landed in HDFS for scalable storage, and processed in scheduled batches with PySpark jobs that compute usage patterns, peak periods, and per-segment summaries. Python scripts tie the workflow together, trigger the processing runs, and export findings as CSV reports that an operations team could consume directly. The intern must think about partitioning, job efficiency, and verifying results rather than just producing numbers. The project builds practical big-data competence in the energy domain: designing a batch pipeline, writing distributed transformations, and delivering trustworthy summary reports, all skills that transfer directly to entry-level data engineering work.

businessenergybig-datahdfs+3

View project

Easy

Minimal REST Serving Layer for Keras Models With FastAPI

Between a trained deep learning model and any product that uses it sits the same component every time: a serving layer that accepts requests, runs inference, and returns predictions. This project builds that component in its cleanest minimal form so every part of it is understood. The intern builds a FastAPI project that loads a saved Keras model at startup and exposes a REST prediction endpoint. The work covers the serving fundamentals with care: structuring the application so the model loads once rather than on every request, defining typed request and response schemas so inputs are validated automatically, and shaping incoming JSON into the arrays the network expects. Worked example requests and responses document the contract precisely, and FastAPI's generated interactive docs make the endpoint testable from a browser — a pattern that applies to e-commerce recommendation calls as readily as to any other inference task. The project distills model serving to its essentials so each concept lands clearly. The intern finishes able to put any Keras model behind a documented HTTP interface — the connective skill that makes deep learning usable by web and mobile applications.

e-commerceeducationdata-sciencedeep-learning+2

View project