In Spark, how can you retrieve the top two highest salaries per department without importing additional functions?

You can use a window function to rank salaries within each department and then filter the top two. For example:

from pyspark.sql import Window
from pyspark.sql.functions import row_number

w = Window.partitionBy("department").orderBy(col("salary").desc())
result = df.withColumn("rank", row_number().over(w))\
           .filter(col("rank") <= 2)\
           .select("department", "salary")

This approach requires no external UDFs and runs efficiently because Spark can push the ranking logic into the execution plan. The complexity is O(n log n) for sorting within each partition, but Spark optimizes this with distributed sorting. If you prefer SQL, you can write:

SELECT department, salary
FROM (
  SELECT department, salary,
         ROW_NUMBER() OVER (PARTITION BY department ORDER BY salary DESC) AS rn
  FROM employees
) t
WHERE rn <= 2;

Both methods produce the same result and are idiomatic Spark solutions.