Have you worked with streaming pipelines and batch networks?

Sure. In my previous role at XYZ, I was responsible for designing a hybrid data ingestion system that combined real‑time streaming and batch processing. The situation was that our e‑commerce platform needed near‑real‑time inventory updates while also running nightly analytics jobs. My task was to integrate Kafka for streaming data and Spark Structured Streaming for real‑time processing, while using Spark batch jobs for nightly aggregations. I set up Kafka topics for order events, used Spark Structured Streaming to consume these topics, performed transformations, and wrote results to a Delta Lake table. For batch, I scheduled nightly Spark jobs that read from the same Delta Lake, performed heavy aggregations, and stored results in a data warehouse. The result was a 30% reduction in data latency for real‑time dashboards and a 25% improvement in batch job throughput, while keeping the architecture maintainable and scalable.