Do you schedule end‑to‑end flows in Databricks, for example ingesting data through an API, performing cleansing and transformations all the way to the core layer? Are these runs scheduled to execute daily or at specific intervals? If failures occur, how do you receive notifications and troubleshoot the issue?

Yes, I use Databricks Jobs to orchestrate the entire pipeline. I define a job that triggers a notebook or a set of notebooks: first an API ingestion notebook, then a cleansing notebook, and finally a core‑layer write notebook. I schedule the job using cron expressions or event‑based triggers (e.g., when new data lands in a storage bucket). For failure handling, I enable job alerts that send emails or Slack messages when a run fails. I also configure retry policies and set up a monitoring dashboard that shows run status, duration, and error logs. When a failure occurs, I quickly inspect the Spark UI and the job logs to pinpoint the error, apply a fix, and re‑run the job. For critical pipelines, I add automated rollback or data validation steps to ensure data integrity.