What is the difference between narrow and wide transformations?

In Spark, transformations are operations that produce a new RDD from an existing one. They are classified as narrow or wide based on data movement. Narrow transformations (e.g., map, filter, flatMap, union) have a one-to-one or many-to-one relationship between input and output partitions. Each output partition depends on a single input partition, so no data shuffling is required; the operation can be performed locally on each partition. Wide transformations (e.g., groupByKey, reduceByKey, join, distinct) involve a many-to-many relationship, meaning each output partition may need data from multiple input partitions. This requires a shuffle phase where data is redistributed across the cluster, involving network I/O, disk writes, and sorting. Consequently, narrow transformations are generally faster and more efficient, while wide transformations are more expensive due to the shuffle overhead.