The catalog

300+ projects to build

Every project comes with a complete PRD, HLD & LLD. Filter by domain, difficulty or tech stack to find your next build.

6 projects

Sort

Easy

Energy Sensor Streaming Dashboard with Amazon Kinesis

Energy companies and smart-building operators deploy fleets of IoT sensors reporting power usage, temperature, and equipment status every few seconds, but the readings only become valuable when they can be seen and acted on as they arrive. Understanding how streaming ingestion works, producers, streams, and consumers, is the entry ticket to every real-time data engineering role. The intern builds a working end-to-end streaming pipeline on AWS. A Python producer simulates IoT sensor devices and writes readings into an Amazon Kinesis Data Stream, a consumer application reads records from the stream as they land, and a dashboard visualizes the incoming measurements with live-updating charts showing values over time and simple aggregates such as averages per sensor. Along the way the intern works with the core Kinesis concepts of streams, shards, and partition keys, and learns how big data systems decouple the components that generate data from the components that use it. Completing the project gives the intern their first real streaming architecture, a producer-consumer system they built themselves rather than a diagram they memorized, along with practical AWS experience and data visualization skills. It is a compact but genuine demonstration that they can move data in real time from source to screen.

energyawsbig-data+3

View project

Hard

Smart Farm Data Platform with Delta Live Tables Orchestration

Modern farms generate relentless data, soil sensors, weather stations, and drone imagery telemetry, but agronomy teams rarely have engineering support to turn those feeds into trustworthy analytics. The gap is architectural: raw sensor streams need ingestion, validation, transformation, and governance before anyone can responsibly base irrigation or planting decisions on them. The intern designs an end-to-end smart agriculture platform on Databricks and Spark. Sensor and drone data arrive through Kafka streams and Autoloader file ingestion, then flow through a Medallion Architecture implemented with Delta Live Tables: Bronze tables capture raw records, Silver tables apply DLT expectations that quarantine bad readings, and Gold tables aggregate field-level metrics. Databricks Workflows and Airflow orchestrate the pipelines on schedules and dependencies, while Databricks SQL dashboards visualize crop health indicators and yield predictions from the curated layer. The intern applies Unity Catalog for governance, secret scopes for credential handling, and cluster policies aimed at cost control. The project demonstrates command of the modern lakehouse stack, streaming plus batch ingestion, declarative pipelines with enforced data quality, dual orchestration tools, and security practices, applied to a domain where data quality has physical consequences.

agricultureairflowbig-data+11

View project

Hard

Governed Streaming Lakehouse for Patient Vitals Monitoring

Hospital wards increasingly stream patient vitals, heart rate, oxygen saturation, blood pressure, from bedside IoT devices, but a raw firehose of readings helps no one: clinicians need validated, trend-aware views, and hospitals need airtight governance over who accesses medical data. Building that refinery for physiological data is a demanding, realistic data engineering challenge. The intern develops a real-time pipeline on Databricks using Spark Structured Streaming and Delta Lake. Device readings stream in from Kafka into Bronze tables, Delta Live Tables enforce schemas and data quality rules that catch malformed or physiologically impossible values, and the Medallion Architecture refines data through Silver cleaning into Gold aggregates per patient and ward. Databricks SQL dashboards visualize vital-sign trends and flag anomalies such as sustained deviations from baseline, giving clinical staff a live monitoring surface. Governance is treated as a first-class requirement: Unity Catalog controls data access, secret scopes protect credentials, and role-based permissions separate engineering from clinical viewers. The project demonstrates streaming pipeline engineering under data quality and privacy constraints, exactly the combination healthcare employers struggle to hire for, and shows the intern can build analytics infrastructure worthy of safety-critical data.

healthcarebig-datadata-analytics+10

View project

Medium

Media Event Stream Analytics with Spark Structured Streaming

Streaming platforms and digital publishers succeed or fail on engagement, and engagement lives in event data: plays, pauses, skips, clicks, and session activity flowing in continuously from apps and players. Product teams need those events aggregated into trustworthy metrics within minutes, not in tomorrow's batch report, which demands a genuinely streaming analytics architecture. The intern constructs a real-time analytics platform on Databricks for media interaction events. User events arrive via Kafka while supplementary files land through Autoloader, and Spark Structured Streaming processes both into Delta Lake tables organized in a Medallion Architecture. DLT pipelines handle the ETL flow with declarative transformations and quality expectations, and the intern implements streaming joins to enrich raw events with content metadata, one of the trickier and most valuable patterns in stream processing. Gold layer tables compute engagement metrics such as views per title, session durations, and completion rates, visualized through Databricks SQL dashboards that refresh as new events arrive. Completing the platform teaches the intern watermarking, streaming state, and join semantics on real event data, plus lakehouse modeling with Python and SQL, demonstrating they can deliver the near-real-time metrics that media businesses actually run on.

mediaautoloaderbig-data+11

View project

Medium

Farm Sensor and Weather Data Pipeline on Delta Lake

Agricultural decisions depend on combining slow-moving data, historical weather records and seasonal yield files, with fast-moving field sensor readings, yet these arrive in different formats on different rhythms. A unified pipeline that treats batch and streaming as one system lets agronomists see current field conditions in the context of history rather than in isolation. The intern creates that unified pipeline on Databricks. Batch ingestion loads CSV and JSON datasets, weather archives and sensor exports, using Spark, while Structured Streaming continuously ingests simulated real-time sensor readings. Both paths write to Delta Lake tables refined through a Medallion Architecture: Bronze preserves raw inputs, Silver standardizes units, timestamps, and sensor identifiers with Python and SQL transformations, and Gold produces analysis-ready aggregates such as daily readings per field joined with weather conditions. Databricks SQL dashboards visualize the curated Gold layer, giving users charted insights into moisture trends, temperature patterns, and anomalies worth investigating. The project teaches the intern how batch and streaming ingestion coexist in one lakehouse, how schema standardization turns messy multi-source data into a coherent model, and how to carry raw files all the way through to dashboards stakeholders can use.

agriculturebig-datacsv+10

View project

Medium

Hospital Sensor Data Pipeline with Medallion Architecture

Bedside monitors and wearable sensors emit patient vitals continuously, and the difference between noise and insight is a pipeline that can clean, contextualize, and aggregate those readings as they stream in. For a data engineer entering healthcare, learning to handle physiological telemetry with rigor is a directly employable skill. The intern builds a streaming pipeline on Databricks that ingests simulated patient vitals, heart rate, temperature, oxygen levels, from Kafka using Spark Structured Streaming. Readings land in a Bronze Delta Lake table with schema enforcement rejecting malformed records, Silver transformations standardize and validate the data with DLT quality rules, and streaming joins enrich readings with patient and device reference data. The Gold layer aggregates vitals into per-patient trends and threshold-based anomaly flags, which the intern visualizes in Databricks SQL dashboards highlighting readings that deviate from expected ranges. Python and SQL implement the transformation logic across all three Medallion layers. Through the build the intern learns structured streaming fundamentals, schema enforcement, watermarks, and stream enrichment joins, within a domain that demands care, and demonstrates the ability to construct a layered, quality-gated pipeline from raw sensor stream to clinical-facing dashboard.

healthcarebig-datadatabricks+10

View project