Explain how repartitioning works in Spark and when it is used.

In Spark, repartition() creates a new DataFrame or RDD with a specified number of partitions by performing a full shuffle of the data. This operation redistributes all rows across the new partitions, ensuring an even distribution and improving parallelism. It is typically used when you need to increase the number of partitions for parallel processing, or when you want to balance data after a filter that skews the distribution. Repartition is also useful before a join or aggregation to avoid data skew. The shuffle cost is O(n), so it should be used judiciously. Coalesce() is the non‑shuffling counterpart used only for reducing partitions.