You explained how you created dependency maps and used metadata queries for dependency. How did you verify the accuracy of these dependency mappings, especially when some dependencies were not automatically captured?

To verify dependency mappings, I first cross‑checked the automated lineage generated by Databricks’ Unity Catalog against the actual execution logs. For each job, I extracted the spark.sql.catalog and spark.sql.history tables to confirm that every source table and view referenced in the code appeared in the lineage graph.

When dependencies were missing—often due to dynamic SQL or external data sources—I performed a manual audit. I parsed the notebook code with a simple regex to capture spark.read.format(...).load(...) statements and compared the list to the automated lineage. I also ran a lightweight test job that materialized each source table into a temporary view and executed the pipeline to see if any runtime errors surfaced.

Finally, I automated this validation by creating a unit test that asserts the count of expected dependencies matches the count in the lineage graph. If a mismatch occurs, the test fails and triggers a notification, ensuring that any new or altered dependencies are captured before production deployment.