What do you know about Spark architecture, and how does it work?

Apache Spark is a distributed data processing engine that follows a master‑worker architecture. The driver program is the master; it parses the user code, creates a logical execution plan, and then transforms it into a physical plan. The cluster manager (YARN, Mesos, or Kubernetes) allocates resources and launches executors on worker nodes. Each executor runs as a JVM process and hosts multiple tasks. Spark’s core abstraction is the Resilient Distributed Dataset (RDD), an immutable distributed collection of objects that can be operated on in parallel. Higher‑level APIs (DataFrames, Datasets) are built on top of RDDs and provide optimizations like Catalyst query planning and Tungsten execution.

Execution proceeds in stages. The logical plan is converted into a Directed Acyclic Graph (DAG) of stages, where each stage contains tasks that can run in parallel. A stage boundary occurs when a shuffle is required (e.g., groupBy, join). Tasks read data from local disk or HDFS, perform transformations, and write intermediate results to disk or memory. Spark’s scheduler assigns tasks to executors, balancing load and handling failures via lineage recovery. Caching and persistence allow intermediate RDDs to be stored in memory for faster reuse. Overall, Spark’s architecture enables fast, fault‑tolerant, in‑memory processing of large datasets across a cluster.