Retail Sales Aggregation Pipeline with Kafka and PySpark

Added Nov 2025 3 design docs

Retailers live and die by their numbers: daily sales by store, weekly trends by category, and the early signals of a product taking off or dying. Producing those numbers reliably from a continuous stream of transactions is a foundational data engineering task, and this project has the intern build the full pipeline behind it. Sales transactions are streamed through Kafka, simulating point-of-sale events flowing in from many stores. The events are stored in HDFS, establishing a durable record of every transaction, and scheduled PySpark batch jobs process the accumulated data to calculate daily and weekly sales summaries, category-level aggregates, and store comparisons. Python scripts manage the pipeline, coordinating ingestion, triggering batch runs, and exporting results as CSV files that a merchandising or finance team could pick up directly. The scope is strictly the data pipeline and its reports, keeping attention on correctness, idempotent processing, and clean output structure. The intern learns the streaming-to-batch architecture used across retail analytics, gains real PySpark aggregation experience, and finishes able to explain how raw transaction events become the numbers a business runs on.

Related projects

Energy Consumption Batch Processing and Reporting Pipeline

Utilities and large facilities collect meter readings from thousands of households and industrial sites, and the questions that matter, such as which sites are trending upward, where peak loads occur, and how weekday usage differs from weekends, require processing volumes of data that outgrow a single machine. Batch analytics on a distributed stack is the industry-standard answer. In this project the intern designs that stack from ingestion to report. Simulated household and industrial consumption records are streamed through Kafka, landed in HDFS for scalable storage, and processed in scheduled batches with PySpark jobs that compute usage patterns, peak periods, and per-segment summaries. Python scripts tie the workflow together, trigger the processing runs, and export findings as CSV reports that an operations team could consume directly. The intern must think about partitioning, job efficiency, and verifying results rather than just producing numbers. The project builds practical big-data competence in the energy domain: designing a batch pipeline, writing distributed transformations, and delivering trustworthy summary reports, all skills that transfer directly to entry-level data engineering work.

businessenergybig-datahdfs+3

View project

Easy

Farm Sensor Data Ingestion Pipeline with Kafka and HDFS

Modern farms generate continuous streams of readings for soil moisture, temperature, humidity, and equipment status, but the raw numbers are useless until they are reliably collected, stored, and summarized. Agritech companies solve this with streaming data platforms, and this project introduces that architecture at a scale a newcomer can build end to end. The intern writes Python producers that simulate realistic agricultural sensor streams and publish them to Kafka topics, then builds consumers that read the streams, validate and enrich the readings, and land them in HDFS for durable, distributed storage. On top of the stored data, Python analysis scripts compute aggregates such as average soil moisture per field, daily temperature ranges, and gaps that indicate sensor failures, producing summary files a farm manager could act on. The focus stays on the pipeline itself: partitioning topics sensibly, handling malformed messages, and organizing HDFS directories for easy downstream analysis. Through this big-data project the intern gains a concrete understanding of publish-subscribe streaming, distributed file storage, and pipeline design, which are foundational skills for any data engineering role.

agriculturetechnologybig-datahdfs+2

View project

Hard

Interactive Knowledge Graph Explorer for Banking Documents

Banks accumulate enormous libraries of reports, contracts, and statements, and the connections buried across those documents, such as which policies reference which entities, or how terms in one contract relate to clauses in another, are nearly impossible to see by reading files one at a time. Analysts and compliance staff need a way to explore that knowledge as a connected map rather than a pile of PDFs. The intern builds an exploration suite in python with a streamlit front end where users upload diverse banking documents. Text is chunked and embedded into a vector database, and a retrieval-augmented generation pipeline finds related passages across the corpus while generative AI produces explanations of how concepts, entities, and policies are linked. The centerpiece is an interactive map of key terms and their relationships that users can query in natural language, asking, for example, how a policy connects to a counterparty mentioned in a different report, with RAG grounding every answer in the source documents. The project is deliberately modular so a team can divide ingestion, retrieval, and interface work, and it leaves the intern with a portfolio-quality demonstration of embeddings, semantic retrieval, and grounded generation applied to a hard, realistic problem.

bankingbusinessgenerative-aipython+4

View project