You're tasked with designing a distributed data processing system using Apache Spark and Apache Kafka that can process 10 million events per second and write the results to a data warehouse. How would you design the system and what technologies would you use?

For 10 M events/s, the architecture starts with a Kafka cluster tuned for high throughput: 3–5 brokers, each with 8–12 partitions per topic, and a replication factor of 3. Producers use Avro with a Confluent Schema Registry to enforce schema and enable compression (Snappy or LZ4). Spark Structured Streaming runs on a YARN or Kubernetes cluster with 200+ executors, each with 4–8 cores and 16 GB RAM. The stream reads from Kafka in micro‑batch mode (e.g., 5 s batches) or continuous mode, applying windowed aggregations. Stateful operations use checkpointing to HDFS or S3 for fault tolerance. Results are written to a columnar store (e.g., Amazon Redshift, Snowflake, or a Hive‑compatible data warehouse) via JDBC or a Kafka Connect sink connector. To handle backpressure, use Kafka’s consumer lag metrics and Spark’s dynamic allocation. For low latency, enable shuffle partitions equal to the number of Kafka partitions. Monitoring is done with Prometheus/Grafana, and alerting on lag or error rates. This design scales horizontally, provides exactly‑once semantics, and keeps the end‑to‑end latency under a few seconds.