What tools did you use and how did you manage dependencies between jobs?

I used Airflow, Luigi, and Dagster to manage job dependencies. In Airflow, I defined tasks as operators and set upstream/downstream relationships using the >> and << operators. This made dependencies explicit and allowed Airflow to schedule tasks only when their prerequisites completed successfully. Luigi’s requires() method served a similar purpose for smaller, Python‑centric pipelines. Dagster’s solids and pipelines provided a type‑safe way to declare dependencies and pass data between stages. For cross‑pipeline dependencies, I used Airflow’s ExternalTaskSensor to wait for a task in another DAG to finish. I also maintained a central metadata store (PostgreSQL) to record job run statuses, enabling downstream jobs to query the state of upstream jobs before execution. This approach ensured clear dependency graphs, easier debugging, and reliable execution order.