Can you explain the architecture of Spark?

Spark’s architecture is built around a driver program and multiple executors that run on a cluster manager (YARN, Mesos, or Kubernetes). The driver creates a logical execution plan (DAG) from the user’s code. The DAG Scheduler breaks this plan into stages based on shuffle boundaries. The Task Scheduler then assigns tasks to executors, which run the actual work. Executors maintain a storage manager that caches RDDs or DataFrames in memory or on disk, and a shuffle manager that handles data exchange between stages. Spark’s core engine uses Tungsten for memory management and whole-stage code generation to optimize CPU usage. The SQL module adds a Catalyst optimizer that rewrites queries for efficiency. Structured Streaming builds on the same execution engine, adding a source and sink abstraction for continuous processing. The cluster manager coordinates resource allocation, while the driver monitors progress and handles failures, ensuring fault tolerance through lineage and recomputation.