Since you haven't shared your unit testing approach yet, how do you perform integration testing in your data pipeline projects to ensure end‑to‑end data quality?

I treat integration testing as a separate pipeline that runs against a sandbox environment. First, I create a set of deterministic test datasets that mimic production schema and cardinality. I then trigger the full pipeline (extract, transform, load) using these datasets. After the pipeline completes, I run a series of data quality checks: schema validation, row‑count assertions, null‑percentage thresholds, and business rule checks (e.g., no negative balances). I use Great Expectations to define expectations declaratively and run them as part of the pipeline. For performance, I compare execution metrics (runtime, resource usage) against baseline thresholds. Finally, I store the test results in a dashboard for auditability. This end‑to‑end approach ensures that any change in upstream or downstream systems is caught before promotion to production.