Imagine the same pipeline ingesting data for a few hours, and one upstream feed starts sending bad or partially corrupted records. How would you design data quality checks and failure handling?

I’d implement a multi-layered validation strategy. First, at ingestion, I’d use schema validation (e.g., JSON schema or Avro) to reject records that don’t match the expected structure. Second, I’d add business rule checks—such as range checks, mandatory fields, and checksum validation—using a framework like Deequ or Great Expectations. Third, I’d route failed records to a dead-letter queue (DLQ) in SQS or Kafka for later inspection. For streaming pipelines, I’d use a watermark to allow late data but still flag anomalies. I’d also set up alerting (CloudWatch alarms) when the error rate exceeds a threshold. Finally, I’d design the downstream jobs to be idempotent and tolerant of missing data, so that a few bad records don’t break the entire pipeline. This combination of schema enforcement, rule-based validation, DLQ, and monitoring provides robust data quality and graceful failure handling.