How do you handle small files in Spark?

Small files in Spark can cause a large number of partitions, leading to high overhead and poor parallelism. To handle them, you can combine small files into larger ones before processing. Techniques include:

1. Use Hadoop's CombineFileInputFormat or Spark's spark.sql.files.maxPartitionBytes to merge small files into larger partitions.
2. Repartition or coalesce the RDD/DataFrame to reduce the number of partitions, e.g., df.repartition(1) for a single large partition or df.coalesce(n) to reduce without shuffling.
3. Store data in a format that supports efficient splitting, such as Parquet or ORC, and enable partitioning on key columns.
4. If the data is already in HDFS, use tools like hadoop fs -getmerge or hadoop fs -cat to merge files.

These steps reduce task overhead, improve I/O efficiency, and lead to better cluster utilization.