What is the difference between repartition and coalesce?

Repartition and coalesce are both used to change the number of partitions in a Spark dataset, but they differ in how they handle data movement. Repartition always performs a full shuffle of the data across the cluster, which can be expensive but guarantees a balanced distribution of data across the new partitions. It can be used to increase or decrease the number of partitions. Coalesce, on the other hand, is a narrow transformation that only reduces the number of partitions without a full shuffle. It simply collapses existing partitions into fewer ones, which is more efficient when decreasing partitions but may lead to uneven data distribution if the original partitions were imbalanced. In practice, use coalesce when you need to reduce partitions (e.g., before writing to a file system) and repartition when you need to increase partitions or require a balanced shuffle.