Directory Watch Ingestion Demo with Databricks Autoloader

Added Jun 2025 3 design docs

A remarkable share of production data pipelines begin with files landing in a storage location, media asset exports, log drops, partner deliveries, and the naive approach of rescanning entire directories collapses as volume grows. Databricks Autoloader exists precisely for this: it discovers new files incrementally and feeds them into streaming pipelines without reprocessing what it has already seen. The intern builds a simple but genuinely real-time pipeline that monitors a directory for arriving files using Autoloader on Databricks. As new files land, Spark ingests them incrementally into a Delta table, with the intern configuring schema inference and watching how Autoloader tracks processed files through its checkpoint state. A dashboard view displays live updates as ingestion proceeds, showing file counts, recent records, and arrival timing, so the streaming behavior is visible rather than theoretical. Python drives the pipeline logic, and the intern experiments by dropping files mid-run to watch them appear downstream within moments. The project teaches incremental ingestion, checkpointing, and streaming triggers in their most approachable form, and gives the intern a working mental model of how modern platforms achieve exactly-once file processing, preparation for every larger streaming architecture they will meet next.

Related projects

Hierarchical Clustering Explorer for News and Media Archives

Editors and content managers at media organizations sit on archives with thousands of articles, transcripts, and posts, and grouping related material by hand simply does not scale. Unsupervised learning offers a way to surface structure in that text without labeling a single document first, and hierarchical clustering is especially useful because it does not force anyone to guess the number of topics in advance. The intern builds a Streamlit application that takes a collection of text documents, converts them into numerical feature vectors, and applies agglomerative hierarchical clustering from scikit-learn to group similar content. SciPy powers the dendrogram rendering, so users can see exactly how documents merge into clusters step by step and choose a sensible cut-off depth visually rather than arbitrarily. The app lets users experiment with different cluster counts and immediately see how the groupings reorganize. The project gives the intern a practical introduction to text vectorization, distance-based clustering, and dendrogram interpretation, along with experience packaging an unsupervised learning workflow into an interactive Python web app that a non-programmer could use to explore a content archive.

mediadata-sciencemachine-learning+4

View project

Easy

Email Spam Detection App Built on Logistic Regression

Spam filtering is one of the oldest applied text classification problems and still one of the best ways to learn the craft: the task is intuitive, the data is text, and mistakes in either direction have obvious costs. This project builds a working spam detector from classical, explainable ingredients. The intern trains a logistic regression classifier with scikit-learn on labeled email data, converting raw text into numerical features through vectorization and using pandas to manage the dataset throughout. A Streamlit interface lets users paste any email text and instantly see the verdict — spam or legitimate — together with the model's confidence, so borderline messages become teachable moments about probability thresholds. The intern experiments with preprocessing choices such as lowercasing and stopword handling, observing directly how each choice moves precision and recall. The project delivers a grounded introduction to text features, linear classifiers, and probabilistic outputs while producing a satisfying interactive artifact. The intern comes away able to explain the full text classification pipeline from raw string to decision — a foundational pattern that scales up to every modern language system.

mediadata-sciencemachine-learning+4

View project

Medium

Document Clustering Web Service With Dendrograms and Linkage Options

Media teams that need to organize large text collections rarely want a notebook — they want a tool: upload documents, choose settings, download organized results. This project builds that tool around hierarchical clustering, the technique of choice when the number of natural groups is unknown in advance. The intern develops a Flask web application where users upload a set of text documents, which are vectorized and clustered with agglomerative methods from scikit-learn. The interface offers a choice of linkage strategies — ward, complete, and average — and renders dendrograms so users can inspect the merge structure and pick a cut point informed by the data rather than a guess. Cluster assignments are exportable as files, with pandas and NumPy managing the data handling, so the results feed directly into downstream editorial or archival workflows. The intern learns text feature engineering, the mechanics and trade-offs of linkage criteria, and dendrogram-driven cluster selection, while practicing web development with Flask — delivering an unsupervised learning capability as a finished, self-service product rather than an analysis that lives on one laptop.

mediadata-scienceflask+5

View project