In Apache Spark, what triggers a shuffle? What is the difference between narrow and wide transformations?

A shuffle is triggered when data must be redistributed across the cluster to satisfy a transformation that requires grouping or partitioning by a key. Typical wide transformations that cause shuffles include groupBy, reduceByKey, join, cogroup, and distinct. They need to move data so that all records with the same key end up on the same executor.

Narrow transformations, such as map, filter, flatMap, and sample, can be executed locally on each partition without moving data. They produce output that stays on the same partition that produced the input. Because no data movement is required, narrow transformations are more efficient and can be pipelined.

Understanding the distinction helps you write jobs that minimize shuffles, which are expensive in terms of network I/O and disk usage.