Can you tell me about the data pipeline you are using in your current project and your role in it?

A typical data pipeline in a modern analytics environment consists of ingestion, transformation, storage, and consumption layers. Ingestion can be batch or streaming; for example, using Kafka or Azure Event Hubs to capture raw logs. Transformation is performed in a compute engine such as PySpark on Databricks, where data is cleaned, enriched, and aggregated. The transformed data is then persisted in a data lake or warehouse (e.g., Delta Lake or Snowflake). Finally, downstream consumers such as BI dashboards or ML models read the curated data. My role involves designing the pipeline architecture, writing PySpark jobs, setting up CI/CD for notebooks, monitoring job health with Databricks Jobs UI, and ensuring data quality through unit tests and schema enforcement. I also collaborate with data scientists to provide feature tables and with DevOps to automate deployments.