An increasing cluster cost: how would you decide between changing partitioning, adjusting join strategy, and scaling the EMR cluster? What evidence would you use to choose the right fix? Additionally, imagine the same pipeline now has to ingest a new source for changing power. With occasional schema drift and the business still wants the load to keep running without manual intervention, how would you design the solution?

First, collect metrics: total job runtime, CPU/memory utilization, shuffle size, and cost per hour. Use these to identify the bottleneck. If the job spends most time shuffling data, consider repartitioning by a more selective key or using bucketing to reduce shuffle. If the join is the culprit, evaluate whether a broadcast join or a map-side join would be cheaper; this is especially true when one side is small. If the cluster is underutilized or tasks are long due to data skew, scaling the cluster (adding more nodes or using spot instances) can reduce cost per job, but only after confirming that the workload is parallelizable. Evidence for each choice comes from Spark UI stages, executor logs, and cost dashboards. For the schema drift scenario, design a pipeline that ingests data into a Delta Lake table with schema evolution enabled. Use a schema registry to validate incoming data against a canonical schema; reject or flag records that violate constraints. Implement automated tests that run on each new source to detect drift early. Finally, orchestrate the pipeline with Airflow or AWS Glue to handle retries and alerting, ensuring the load continues without manual intervention.