How would you design a pipeline that runs whenever there is a change in the data, producing a real‑time report as soon as possible?

A common approach is to use change data capture (CDC) to detect changes in the source system and push them to an event bus such as Kafka or AWS Kinesis. Each change event is processed by a lightweight consumer (e.g., a Lambda function or a microservice) that updates a materialized view or a reporting table in a data warehouse. The consumer writes the transformed data to a fast storage layer (S3 or a cache) and then triggers a downstream job that aggregates the data and generates the report. To ensure low latency, the consumer can write directly to the reporting layer using an upsert operation, and the reporting layer can expose the data via a real‑time query engine like Amazon Athena or Redshift Spectrum. The overall latency is bounded by the CDC capture interval, the event bus propagation time, and the consumer processing time, typically keeping it under a few seconds.