What join configuration optimization technique would you apply to fix crashes without increasing memory usage?

When a join job crashes but memory usage is not the culprit, the issue often lies in how data is partitioned, shuffled, or processed. A common optimization is to use a broadcast (map‑side) join when one side of the join is small enough to fit in memory. This eliminates the shuffle phase entirely, reducing disk I/O and network traffic. If the small table is still too large, you can apply a semi‑join or filter‑pushdown to reduce the amount of data that needs to be shuffled. Another technique is to adjust the number of reducers or map tasks to better match the cluster’s parallelism, ensuring that no single task becomes a bottleneck. You can also enable compression on shuffle files (e.g., Snappy or LZO) to reduce disk usage and network transfer. Finally, tuning the join algorithm (e.g., using a sort‑merge join instead of a hash join when data is already sorted) can improve performance without increasing memory. These steps address data movement and processing efficiency, which are often the root causes of crashes unrelated to memory.