Please give a brief introduction about yourself and describe your experience in developing and monitoring ETL pipelines using Grafana and cloud-based technologies. Explain the stages involved in developing such a pipeline.

Situation: In my previous role at a retail company, we had fragmented data sources across on-prem and cloud systems.

Task: I was tasked with building a unified ETL pipeline to consolidate sales, inventory, and customer data for analytics.

Action: I designed the pipeline in three stages:

1. Extraction – used Airflow to schedule daily pulls from APIs and database dumps, storing raw files in S3.
2. Transformation – wrote Spark jobs to clean, deduplicate, and enrich data, writing results to a data lake.
3. Loading – loaded the cleaned data into Redshift for reporting.

I set up Grafana dashboards to monitor job status, latency, and error rates, and configured alerts for failures.

Result: The pipeline reduced data latency from 48 hours to 4 hours, improved data quality, and enabled real-time dashboards for business users. The monitoring system cut incident response time by 70%.