You mentioned enabling Adaptive Query Execution (AQE) and query execution. Can you explain how AQE helped improve performance in your Spark job, particularly with respect to join keys?

Adaptive Query Execution rewrites the physical plan at runtime based on actual statistics collected during execution. For joins, AQE can dynamically choose between broadcast, shuffle‑hash, or shuffle‑merge join strategies after measuring the size of the broadcastable side. In my last project, a join between a 10‑GB fact table and a 200‑MB dimension table was initially executed as a shuffle‑merge join, causing a 30‑minute shuffle. After enabling AQE, Spark measured the dimension table size, recognized it was small enough to broadcast, and automatically switched to a broadcast‑hash join, cutting the runtime to 5 minutes. AQE also adjusts the number of shuffle partitions to match the actual data size, reducing GC pressure and improving parallelism. The key takeaway is that AQE removes the need to manually tune join strategies and partition counts, letting Spark adapt to real data characteristics.