How would you identify bottlenecks and optimize Spark performance? Can you walk me through your approach?

My approach starts with the Spark UI and the event timeline. I look for stages with the longest duration, high GC overhead, or skewed task times. If a stage shows many long tasks, I suspect data skew and will repartition or use salting. For shuffle‑heavy jobs, I check the shuffle read/write metrics; high shuffle bytes indicate a need for better partitioning or broadcast joins. I also monitor executor memory usage and CPU utilization; if executors are spilling, I increase memory or reduce the number of partitions. Next, I enable Tungsten and Catalyst optimizations by ensuring the Spark configuration has spark.sql.shuffle.partitions tuned to the cluster size and that spark.sql.autoBroadcastJoinThreshold is set appropriately. I also cache intermediate DataFrames that are reused. Finally, I profile the code to identify expensive UDFs, replace them with native Spark functions, and use vectorized UDFs if necessary. After each change, I rerun the job and compare the metrics to confirm improvement.