How is the physical plan created in Spark, and how are stages, tasks, and jobs generated?

When a Spark application runs, the driver first parses the user code into a logical plan. The Catalyst optimizer then transforms this logical plan into an optimized logical plan. Next, the physical planner chooses a physical plan that specifies how to execute each operator (e.g., hash‑join, sort‑merge‑join). This physical plan is a DAG of stages. A stage is a set of tasks that can be executed in parallel and has a single shuffle boundary. Each stage is split into tasks, one per partition of the input RDD or DataFrame. A job is triggered by an action (e.g., count, collect). The job is broken into stages based on shuffle dependencies. Spark schedules tasks within a stage on executors; when all tasks of a stage finish, the next stage begins. Thus, stages, tasks, and jobs are derived from the logical plan, optimized into a physical plan, and then partitioned by shuffle boundaries.