Have you heard about broadcast join?

A broadcast join is a join strategy in distributed data processing frameworks like Apache Spark where one of the tables (usually the smaller one) is broadcasted to all worker nodes. This eliminates the need for shuffling the larger table across the network, reducing I/O and network overhead. It is most effective when the broadcasted table fits comfortably in the memory of each executor. For example, joining a large user table with a small lookup table of user attributes can be done efficiently by broadcasting the lookup table. The cost of broadcasting is the memory usage on each executor, so it should be used only when the broadcasted dataset is small enough to avoid memory pressure.