Imagine an AWS pipeline that now has to serve an analytics team needing low-latency SQL access on curated data, while the source tables keep changing schema over time. How would you design the ingestion and transformation layer to handle schema evolution safely while keeping the SQL layer stable?

I’d use AWS Glue Data Catalog as a central schema registry and enable schema evolution on Glue crawlers. The ingestion layer would read from Kinesis or S3, transform data with Glue ETL jobs or Spark on EMR, and write to a Delta Lake or Iceberg table on S3. Both formats support schema evolution and ACID transactions. The transformation job would detect schema changes via the catalog, apply ALTER TABLE statements to add new columns with default values, and maintain backward compatibility. For the analytics layer, I’d expose the curated table through Amazon Athena or Redshift Spectrum, which can query the underlying Delta/Iceberg files. To keep the SQL layer stable, I’d create a view layer that maps the evolving schema to a fixed set of columns, using COALESCE for missing fields. Additionally, I’d implement automated tests that run after each schema change to validate query results. This approach decouples source evolution from downstream consumers, ensuring low-latency access and minimal disruption.