In Spark, there is something called partition and repartition. Can you explain the difference between them?

In Spark, a partition is a logical division of an RDD or DataFrame that determines how data is distributed across the cluster. Each partition is processed by a single executor task. The number of partitions is set at creation time or can be changed later. Repartition is an action that creates a new RDD/DataFrame with a different number of partitions. It triggers a full shuffle of the data across the cluster, which can be expensive but allows you to balance data or increase parallelism. Repartition can increase or decrease the number of partitions. In contrast, coalesce can reduce the number of partitions without a full shuffle, but it cannot increase them. Therefore, repartition is used when you need to change the partition count in either direction and are willing to incur the shuffle cost.