How would you partition data and on what basis? If you are going to change the file size, what is partitioning? Also, from an ETL perspective, where would you run a Lambda function to process a 20GB file? Which cluster would it run on?

Partitioning is a way to split a dataset into manageable chunks. I would partition by a logical key such as date, region, or user ID, ensuring each partition is roughly the same size (e.g., 100–200 MB) to balance parallelism and I/O. If file size changes, I’d adjust the partition key or use bucketing to keep partitions balanced. For a 20 GB file, AWS Lambda is unsuitable because its maximum memory is 10 GB and the execution timeout is 15 minutes, which would likely exceed limits. Instead, I would use an EMR cluster or Glue ETL job, which can run Spark or Python scripts on a distributed cluster. Glue can automatically create a crawler to catalog the data, and EMR can scale to handle the file size, using S3 as the source and destination. If real‑time processing is needed, I could split the file into smaller chunks, upload them to S3, and trigger a Lambda for each chunk, but for bulk ETL, EMR/Glue is the right choice.