What was the project about? What type of data were you receiving?

The project was a real‑time data ingestion and analytics platform for an e‑commerce company. Our goal was to capture user interactions, process them, and provide actionable insights for marketing and product teams. We received a mix of structured and semi‑structured data: JSON logs from the front‑end, CSV files from partner APIs, and binary images for product catalog updates. The ingestion layer used Apache Kafka to buffer high‑velocity streams, while Apache Spark performed batch and streaming transformations to clean, enrich, and aggregate the data. Processed data was stored in an Amazon S3 data lake for long‑term retention and in Amazon Redshift for fast analytical queries. We also implemented a monitoring stack with Prometheus and Grafana to track pipeline health and latency. This architecture allowed us to scale horizontally, handle spikes in traffic, and deliver near‑real‑time dashboards to stakeholders.