What happens if a Spark job fails at 50% completion?

Spark’s fault tolerance is built on lineage. If a task fails, Spark automatically retries it on the same or a different executor. If an executor dies, all tasks on that executor are rescheduled. At 50% completion, the driver will detect the failure, trigger a reschedule, and recompute only the lost partitions, not the entire job. If the failure is due to a persistent issue (e.g., data corruption), the job may abort. Speculative execution can mitigate slow tasks, and checkpointing can reduce recomputation cost for long lineage chains. In a cluster manager like YARN or Kubernetes, the driver may also be restarted if it crashes, leading to a full job restart. Proper configuration of spark.task.maxFailures, spark.speculation, and checkpointing can control how aggressively Spark recovers from failures.