Why do we need to use Spark? Why can't we directly perform computations using Python itself?

Python is great for prototyping and small data sets, but it runs in a single process and is limited by the memory of that process. Spark, on the other hand, is a distributed data processing engine that splits data across many nodes, schedules tasks in parallel, and recovers from node failures. Spark’s Resilient Distributed Datasets (RDDs) and DataFrames allow lazy evaluation, in‑memory caching, and automatic query optimization. These features give Spark a performance advantage for large‑scale analytics, iterative machine‑learning pipelines, and real‑time streaming. Additionally, Spark integrates with Hadoop, Hive, Cassandra, and cloud storage, enabling seamless data ingestion and export. In short, Spark is used when the problem size, fault tolerance, and scalability requirements exceed what a single Python process can handle.