Hospital Sensor Data Pipeline with Medallion Architecture
Bedside monitors and wearable sensors emit patient vitals continuously, and the difference between noise and insight is a pipeline that can clean, contextualize, and aggregate those readings as they stream in. For a data engineer entering healthcare, learning to handle physiological telemetry with rigor is a directly employable skill.
The intern builds a streaming pipeline on Databricks that ingests simulated patient vitals, heart rate, temperature, oxygen levels, from Kafka using Spark Structured Streaming. Readings land in a Bronze Delta Lake table with schema enforcement rejecting malformed records, Silver transformations standardize and validate the data with DLT quality rules, and streaming joins enrich readings with patient and device reference data. The Gold layer aggregates vitals into per-patient trends and threshold-based anomaly flags, which the intern visualizes in Databricks SQL dashboards highlighting readings that deviate from expected ranges. Python and SQL implement the transformation logic across all three Medallion layers.
Through the build the intern learns structured streaming fundamentals, schema enforcement, watermarks, and stream enrichment joins, within a domain that demands care, and demonstrates the ability to construct a layered, quality-gated pipeline from raw sensor stream to clinical-facing dashboard.
