You are orchestrating a 5‑task lakehouse workflow where Task A discovers a partition path (e.g., s3://raw/yyyyymmdd=20231015) that downstream tasks must use. Finance wants ephemeral, policy‑restricted compute and parallel fan‑out where possible. Which Databricks pattern best fits?
💡 Model Answer
The most suitable pattern is to run all five tasks on a single job cluster and share the discovered partition path via a temporary Delta table. This approach keeps compute transient (the cluster is terminated after the job), satisfies policy restrictions, and allows downstream tasks to read the path from the Delta table. Because all tasks run on the same cluster, they can fan‑out in parallel without the overhead of spinning up separate clusters, and the temporary Delta table provides a lightweight, durable way to pass the path between tasks.
This answer was generated by AI for study purposes. Use it as a starting point — personalize it with your own experience.
🎤 Get questions like this answered in real-time
Assisting AI listens to your interview, captures questions live, and gives you instant AI-powered answers — invisible to screen sharing.
Get Assisting AI — Starts at ₹500