Got it. So you're using Git for version control and code reviews. Can you tell me more about how you test data transformations in your CI/CD pipeline? Do you write unit tests, integration tests, or both, and what frameworks do you use?

In our CI/CD pipeline, we test data transformations at two levels. For unit tests, we use pytest to validate individual transformation functions, mocking external dependencies and asserting that given input rows produce the expected output schema and values. For integration tests, we spin up a lightweight data warehouse instance (e.g., Snowflake sandbox or Postgres) using Docker Compose, load sample raw data, run the full ETL job, and then query the fact and dimension tables to verify row counts, key integrity, and business logic. We also use dbt’s built‑in test framework to assert constraints such as uniqueness, non‑null, and referential integrity. Great Expectations is employed for data quality checks on raw streams, ensuring column types, value ranges, and distribution metrics. These tests are triggered in GitHub Actions: a pull request runs unit tests, while a nightly pipeline runs integration tests and dbt tests. If any test fails, the pipeline stops and the code review is blocked, ensuring that only validated transformations reach production.