The catalog

300+ projects to build

Every project comes with a complete PRD, HLD & LLD. Filter by domain, difficulty or tech stack to find your next build.

5 projects

Sort

Hard

Student Clickstream Analytics on the Hadoop Ecosystem

Online learning platforms capture every click, video view, and quiz attempt, signals that could flag struggling students within hours, yet most institutions still analyze engagement weeks later, after the moment for intervention has passed. Acting on learning data in real time requires an architecture that handles both continuous streams and heavyweight historical analysis. The intern engineers that system on the open-source big data stack. Clickstream and assessment events are ingested through Kafka and Flume, Spark Streaming processes them into near-real-time engagement metrics while batch Spark jobs compute deeper historical analyses, and results persist to HDFS with Hive providing warehouse-style querying. Airflow orchestrates the interlocking batch and streaming workflows with dependencies, retries, and schedules, and Presto enables fast federated queries across the stored datasets for ad-hoc questions. Prometheus scrapes pipeline and cluster metrics, with Grafana dashboards displaying both system health and live student engagement indicators such as session activity and assessment completion trends. The project demonstrates rare breadth across the Hadoop-ecosystem toolchain, dual-mode processing, orchestration, federated querying, and observability, proving the intern can integrate many moving parts into one dependable analytics platform.

educationairflowbig-data+10

View project

Hard

Hybrid Batch and Streaming Analytics for E-Commerce Events

E-commerce businesses ask two kinds of questions of the same data: what is happening on the site right now, and what patterns emerge over months of orders. Answering both requires a platform that marries streaming clickstream processing with heavyweight batch analytics over transactional history, coordinated so the two views stay consistent. The intern develops this hybrid platform on the open-source big data stack. Sqoop performs batch ingestion of transaction data from relational databases while Kafka carries real-time clickstream events; Spark handles both modes, with Spark Streaming computing live metrics like active sessions and conversion funnels and batch jobs building historical aggregates. HDFS provides durable storage, Hive exposes warehouse tables over the curated data, and Presto serves fast interactive queries for analysts. Airflow orchestrates the ETL workflows end to end with scheduling and dependency management, and the intern instruments the whole system with Prometheus metrics and Grafana dashboards tracking both cluster health and pipeline throughput. Completing the platform proves the intern can reason about lambda-style architectures, operate the classic Hadoop-ecosystem toolset that large retailers still run at scale, and treat monitoring as part of the deliverable rather than an afterthought.

e-commerceairflowbig-data+10

View project

Hard

Real-Time Streaming Pipeline for Energy Sector Sensor Data

Energy companies operate turbines, substations, and smart meters that emit telemetry every second, but the value of those readings evaporates when they take hours to reach analysts. Grid operators need infrastructure that ingests high-throughput sensor streams, detects problems as they happen, and still keeps a queryable history for engineers and planners. The intern designs and implements that pipeline end to end. Kafka handles high-volume ingestion from simulated sensor fleets, Spark Streaming applies windowed aggregations and fault-tolerant transformations in Python, and results land in HDFS for durable distributed storage. Hive exposes the data as queryable tables while Presto enables fast federated analytics across sources, and Airflow orchestrates the whole workflow with dynamic DAG scheduling, retries, and backfills so the pipeline runs reliably without hand-holding. Along the way the intern learns the core patterns of modern streaming architecture: fault-tolerant processing, watermarking and windowing, schema design for time-series telemetry, and production-grade orchestration. These are the skills that transfer directly to data engineering roles in energy, IoT, and any industry where data arrives as a stream.

energyairflowbig-data+6

View project

Medium

Clickstream Analytics Pipeline for E-Commerce Web Traffic

Every page view, add-to-cart, and video play on a busy e-commerce or media site is a signal, but by the time traditional batch reports surface those signals the campaign is over or the customer is gone. Product and marketing teams need to see traffic behavior within seconds so they can react to spikes, broken funnels, and trending content while it still matters. The intern builds a complete real-time clickstream system in Python. Web server logs flow into Kafka topics that absorb traffic bursts without dropping events, and Spark Streaming jobs parse, sessionize, and aggregate the stream, computing metrics like page views per minute, conversion funnels, and top content over sliding windows. Processed results are written to Hive tables for durable, queryable history, Presto provides low-latency ad-hoc SQL across it all, and a dashboard visualizes live traffic so non-technical stakeholders can watch the site breathe. The project teaches the ingestion-processing-serving pattern used by real analytics teams: partitioning Kafka topics, managing streaming state, designing Hive schemas for time-series events, and exposing insight through both SQL and visualization.

e-commercemediahivekafka+3

View project

Medium

Streaming Anomaly Detection for Industrial Sensor Networks

A single failing pump or overheating transformer can cost a manufacturer or utility far more than the sensor that would have flagged it. The hard part is scale: thousands of devices emit readings continuously, and spotting the one abnormal pattern in that flood requires streaming infrastructure, not spreadsheets and morning reports. In this project the intern builds a real-time analytics platform for industrial telemetry. Kafka ingests high-frequency sensor readings from simulated devices, and Spark Streaming jobs written in Python compute windowed aggregations over the live stream, including rolling averages, min-max bands, and rate-of-change metrics. Spark ML models trained on historical data score incoming readings for anomalies, raising alerts when equipment drifts outside learned norms, while HDFS stores the complete history for replay, auditing, and periodic model retraining. The intern comes away understanding both halves of applied streaming machine learning: the engineering side, with partitioning, checkpointing, and fault-tolerant state, and the data science side, with feature windows, anomaly thresholds, and honest evaluation of detection quality. That combination is in high demand across energy and manufacturing.

energymanufacturinghdfskafka+3

View project