What are the differences between a broadcast join and a normal (shuffle) join in Spark?

A broadcast join replicates the smaller DataFrame to every executor, eliminating the need for a shuffle. This is efficient when one side of the join is small enough to fit in executor memory (typically below the autoBroadcastJoinThreshold). A normal shuffle join partitions both DataFrames on the join key and shuffles data across the cluster, which is expensive in terms of network I/O and disk usage. Broadcast joins reduce shuffle overhead but consume executor memory; if the broadcasted DataFrame is too large, it can cause OOM errors. Normal joins are more scalable for large join partners but can be slower due to the shuffle. Choosing between them depends on the size of the join partners, available memory, and performance requirements.