Suppose you are creating ETL pipelines that perform incremental loads. You have access to an S3 bucket where files are placed at regular intervals. How would you design the pipeline so that it automatically triggers on the latest file without performing a full load?

A common pattern is to use S3 event notifications to trigger an AWS Lambda function whenever a new file lands in the bucket. The Lambda writes a message to an SQS queue or triggers an AWS Glue job. The Glue job (or Spark job) reads only the new file, using the file path or a timestamp column to filter records that have not yet been processed. To avoid reprocessing, maintain a metadata table (e.g., in DynamoDB or a Hive metastore) that stores the last processed timestamp or file name. The job queries the metadata, reads only files newer than that value, performs the transformation, writes results to the target, and updates the metadata. This approach ensures incremental loading, automatic triggering, and avoids full scans of the dataset.