Big Data Final Project
Steam Spam/Bot Review Detection System
MY ROLE
System Architect & Lead Developer
TECHNOLOGY
TIMELINE
About the Project
Architected a hybrid data lakehouse system (Kafka, PySpark, Trino) to process 6.4M+ reviews. Engineered an ML pipeline (Isolation Forest, K-Means, BERT) and built a FastAPI/Streamlit dashboard for real-time visualization.
Key Contributions
Hybrid Architecture Design
Architected a complete data lakehouse system integrating real-time ingestion with Kafka, batch preprocessing with PySpark, and distributed SQL querying using Trino.
Streaming & Ingestion Layer
Developed Kafka producers and consumers for real-time ingestion of over 6.4 million Steam reviews, implementing a raw and clean layer storage strategy directly in MinIO.
Advanced ML Pipeline (Unsupervised & BERT)
Engineered an ensemble anomaly detection system using Isolation Forest and K-Means clustering for auto-labeling, followed by a Hybrid BERT Classifier fused with engineered linguistic features to predict spam probability with high confidence.
API, Dashboard & Orchestration
Developed a FastAPI backend to serve model inferences and a Streamlit/React dashboard for real-time visualization. Created an automated orchestration script (run_all.sh) to synchronize the deployment of 9+ containerized services.