Have you worked on PySpark or Scala?

Situation: In my previous role, I was tasked with building an ETL pipeline to ingest and transform log data for a real‑time analytics platform. Task: I needed to process terabytes of data efficiently while ensuring low latency for downstream dashboards. Action: I used PySpark for rapid prototyping and data exploration, leveraging its DataFrame API for schema inference and built-in functions. For performance‑critical stages, I rewrote critical transformations in Scala, taking advantage of Spark’s native JVM execution and better type safety. I also implemented broadcast joins for small dimension tables and tuned shuffle partitions to match the cluster size. Result: The pipeline processed 5 TB of data in under 30 minutes, reduced memory usage by 20%, and enabled near‑real‑time reporting. This experience demonstrates my proficiency in both PySpark and Scala, and my ability to choose the right tool for each part of the workflow.