Suppose you have a DataFrame and you want to run an SQL query on that DataFrame. How would you do it?

In Spark you can run SQL queries on a DataFrame by registering it as a temporary view and then using SparkSession.sql. For example:

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
# Assume df is your DataFrame

# Register the DataFrame as a temporary view

df.createOrReplaceTempView("my_table")

# Run an SQL query
result = spark.sql("SELECT col1, COUNT(*) FROM my_table GROUP BY col1")
result.show()

Alternatively, if you are using pandas, you can use the pandasql library or convert the DataFrame to a Spark DataFrame and then use the same approach. The key idea is to expose the DataFrame as a table-like abstraction so that Spark’s Catalyst optimizer can plan the query efficiently. Complexity is O(n) for the scan plus any aggregation cost, and Spark handles parallelism automatically.