When building a data pipeline on AWS, you notice row-bound mismatches in a downstream audit table even though the job completes successfully. How do you identify and resolve distributed processing issues in Spark, and what checks do you implement to catch dirty data?

First, I would use the Spark UI and logs to locate the stage where the mismatch occurs. The UI shows task metrics such as input size, shuffle read/write, and task duration. If a particular partition is much larger or slower, it indicates data skew. I would then examine the source data for outliers or missing values that could cause the skew. To catch dirty data, I implement schema enforcement and validation rules using tools like Deequ or Great Expectations. These frameworks allow me to define checks such as non-null constraints, value ranges, and uniqueness. I also add a checksum or hash column to the source data and compare it after the transformation to ensure row integrity. For distributed processing problems, I adjust the number of shuffle partitions, use broadcast joins for small tables, and apply salting or repartitioning to balance load. Finally, I add unit tests that run on a sample dataset to verify that the pipeline produces the expected row count and that no mismatches occur. This combination of monitoring, validation, and tuning ensures that the pipeline remains reliable even with dirty source data.