In a project pipeline that processes data files from various sources, can we replace AWS Glue with AWS Lambda functions?

You can replace Glue with Lambda for simple, event‑driven transformations that process small to medium files (e.g., < 100 MB) and run within 15 minutes. Lambda functions can be triggered by S3 events, transform the data, and write the result back to S3 or another store. However, for complex ETL tasks that involve schema discovery, incremental loading, job scheduling, or processing large volumes of data, Glue remains the better choice. A hybrid approach is often effective: use Lambda for lightweight, real‑time transformations and Glue for heavy, scheduled ETL jobs. This leverages Lambda’s low‑latency, pay‑per‑execution model while retaining Glue’s managed ETL capabilities for larger workloads.