Can you dig deeper and explain what technologies you have used, what processes you follow, and what kind of data transformations you have dealt with on a daily basis?

In my recent role as a data engineer, I primarily work with a Python-based stack. I use Apache Airflow for orchestration, ensuring that data pipelines run reliably and on schedule. For data ingestion, I rely on Kafka to stream real‑time events and AWS S3 for batch uploads. The transformation layer is built on PySpark, which allows me to perform large‑scale data cleaning, deduplication, and aggregation efficiently. I also use dbt for data modeling, which helps maintain a clear lineage and version control for SQL transformations.

Daily tasks involve monitoring DAG runs, troubleshooting failures, and optimizing Spark jobs by tuning partition sizes and memory settings. I frequently work with semi‑structured JSON and CSV files, converting them into a normalized relational schema in Snowflake. Data transformations include type casting, handling nulls, computing derived metrics, and generating summary tables for downstream analytics. I also write unit tests with Great Expectations to validate data quality before loading into the warehouse. This end‑to‑end process ensures that the data pipeline is robust, maintainable, and delivers clean, actionable data to business users.