Security Event Log Pipeline for Suspicious Pattern Detection

Security teams drown in logs: authentication events, network flows, and application errors arrive from every system at once, and the signal of an attack hides inside millions of routine entries. The infrastructure that collects those events at scale and mines them for suspicious patterns is core to modern security operations, and this project has the intern build it. Simulated security event logs from multiple sources are ingested in real time through Kafka, providing a realistic multi-stream feed. Events land in HDFS for scalable retention, and batch PySpark jobs analyze the accumulated logs to detect suspicious patterns, such as repeated failed logins followed by success, activity at anomalous hours, and bursts from single origins, and to generate summary reports for review. Python scripts orchestrate the pipeline and automate the data engineering tasks around it. The deliverables are data artifacts and reports only, keeping the project honest about its infrastructure focus with no web or mobile components. The intern gains hands-on big-data experience framed by security monitoring: streaming ingestion, distributed storage, detection logic in PySpark, and the modular pipeline design that security-scale data demands.