Suppose we are re-ingesting some source data that is about 100 to 200 GB in size and we call the collect() function. What problems can arise from this?

The collect() action pulls all rows from the DataFrame into the driver program’s memory. When the dataset is 100–200 GB, the driver will almost certainly run out of memory, causing an OutOfMemoryError or severe garbage collection pauses. Even if the driver has enough RAM, the network transfer of such a large payload is slow and can become a bottleneck, leading to long job times and potential timeouts. Additionally, the driver’s JVM may become unresponsive, affecting other jobs. To avoid these issues, use actions that keep data on executors, such as write to storage, use take() or limit() for small samples, or process data in partitions with foreachPartition(). If you must bring data to the driver, ensure the dataset is truly small or use a broadcast variable for a small lookup table instead of collect().