Describe how you build an end‑to‑end data pipeline.

Building an end‑to‑end data pipeline involves several key components and design decisions:

1. Source Layer – Identify all data sources (databases, APIs, event streams). For batch sources, schedule extraction jobs; for streaming sources, use Kafka or Kinesis.
2. Ingestion Layer – Use a lightweight orchestrator (Airflow, Prefect) for batch jobs and a stream processor (Kafka Streams, Flink) for real‑time data. Store raw data in a durable, immutable lake (S3, HDFS) with versioning.
3. Processing Layer – Transform data using Spark or Flink for large‑scale transformations. Apply schema evolution handling, data quality checks, and enrichment with reference data. Persist processed data in a data warehouse (Snowflake, BigQuery) or a columnar store (Redshift, ClickHouse).
4. Metadata & Lineage – Capture schema, lineage, and data quality metrics using tools like Amundsen or DataHub. This aids governance and debugging.
5. Monitoring & Alerting – Instrument jobs with metrics (latency, throughput) and set up alerts for failures or SLA breaches. Use Prometheus/Grafana or cloud‑native monitoring.
6. Security & Compliance – Encrypt data at rest and in transit, enforce role‑based access, and audit logs. Implement data masking for sensitive fields.
7. Deployment & CI/CD – Store pipeline code in Git, use CI pipelines to run tests, and deploy via container orchestration (Kubernetes) or serverless runtimes.

Key decisions include choosing batch vs streaming based on latency requirements, selecting a data lake vs lakehouse architecture, and determining the level of schema enforcement. By modularizing the pipeline and using robust tooling, we achieve scalability, maintainability, and data quality across the entire flow.