How was PySpark deployed within the AWS environment, and what services were used?

PySpark can be deployed on AWS in several ways, each suited to different workloads:

1. Amazon EMR (Elastic MapReduce) – You spin up an EMR cluster with Spark installed. PySpark scripts run as YARN applications. This is ideal for batch processing and interactive notebooks via Jupyter.
2. EMR Serverless – A fully managed, on‑demand Spark service where you submit PySpark jobs without provisioning clusters. It scales automatically and charges per second of compute.
3. AWS Glue ETL – Glue provides a serverless Spark runtime. You write PySpark scripts in the Glue console or via Glue Studio, and Glue handles job scheduling, monitoring, and the Data Catalog.
4. Amazon SageMaker Processing Jobs – For ML pipelines, you can run PySpark in a SageMaker processing job, leveraging SageMaker’s managed infrastructure.
5. Amazon EKS with Spark Operator – For Kubernetes‑centric workloads, deploy Spark on EKS using the Spark Operator, which manages Spark applications as Kubernetes custom resources.

Common services used across these options include S3 for data storage, IAM for fine‑grained permissions, CloudWatch for logs and metrics, and the Glue Data Catalog for metadata. The choice depends on cost, scalability, and operational preferences. Complexity of a PySpark job is typically O(n) for data transformations, with additional overhead for shuffling and partitioning.