A group of Spark jobs scanning two terabytes of data crashed because YARN containers exceeded memory limits. How would you address this issue?

First, I would inspect the Spark UI and YARN logs to confirm that the executor memory was insufficient. I would then increase the executor memory and driver memory using spark.executor.memory and spark.driver.memory, ensuring that the total memory does not exceed the container limits. Enabling dynamic allocation (spark.dynamicAllocation.enabled=true) allows Spark to request more executors when needed. I would also reduce the number of partitions with coalesce or repartition to avoid creating too many small tasks that each consume memory. If shuffles are heavy, I would increase spark.shuffle.file.buffer and enable compression. Finally, I would monitor the job after these changes to verify that the containers no longer exceed limits and that the job completes successfully.