How would you optimize a job that processes large data sets and is experiencing high shuffle and low compute?

When a Spark job shows high shuffle and low compute, the bottleneck is usually data movement rather than CPU usage. First, examine the shuffle size by inspecting the job’s stages in the UI; if the shuffle output is large, consider repartitioning the RDD/DataFrame to a higher number of partitions so that each task handles a smaller chunk of data, which reduces the amount of data each reducer must merge. Use repartition() or coalesce() appropriately: repartition() for a full shuffle to a new number of partitions, coalesce() for a narrow shuffle when reducing partitions. Next, enable data locality by ensuring that the data is stored on the same nodes where the tasks run; this can be achieved by persisting the intermediate RDD with MEMORY_AND_DISK and using cache() before heavy transformations. For joins, use broadcast joins when one side is small, which eliminates shuffle entirely. Also, filter early in the pipeline to reduce the amount of data that needs to be shuffled. Finally, tune the shuffle buffer size (spark.shuffle.file.buffer) and the number of shuffle partitions (spark.sql.shuffle.partitions) to match the cluster’s resources. These steps reduce network I/O, balance load across executors, and improve overall throughput.