Why would I use a UDF?

A User‑Defined Function (UDF) lets you extend Spark’s built‑in functions with custom logic written in Scala, Python, or Java. Use a UDF when you need to perform operations that aren’t available in Spark’s native API, such as complex string manipulation, custom aggregation, or domain‑specific calculations. UDFs are applied to DataFrame columns and can be used in select, filter, or groupBy clauses. However, they can be slower than native functions because they break Spark’s optimization engine (Catalyst) and may require serialization overhead, so they should be used sparingly and only when necessary.