How did you handle Hadoop cluster tuning for a big data job that was running slower than expected?

First, I collected job metrics from YARN and Spark UI to identify bottlenecks—CPU, memory, disk I/O, or network. I increased the number of executors and adjusted executor memory to reduce GC overhead, ensuring each executor had enough heap but not too much to cause spill. I tuned the shuffle partitions to match the data size, using spark.sql.shuffle.partitions and adjusting the number of reducers. For skewed joins, I used broadcast joins for small tables and applied salting for large keys. I also enabled compression for shuffle files and tuned the block size in HDFS. Finally, I monitored the job after each change, iterating until the execution time dropped by 30–40%. The key was to use metrics-driven tuning and incremental changes rather than guessing.