What is the difference between repartition and coalesce in Spark, and when would you use each?

Repartition and coalesce are both used to change the number of partitions in a Spark DataFrame or RDD. Repartition performs a full shuffle of the data across all partitions, which can increase or decrease the number of partitions and results in a balanced distribution. It is useful when you need more parallelism or when you want to redistribute data after a filter that skews the data. Coalesce, on the other hand, only reduces the number of partitions and does not shuffle data; it simply merges existing partitions. This is efficient for reducing partitions after a filter or when you want to avoid the cost of a shuffle. Use repartition when you need to increase partitions or balance data, and use coalesce when you only need to reduce partitions without incurring a shuffle.