How would you design an incremental load in a standard data pipeline?

An incremental load extracts only new or changed data since the last run. First, identify a reliable change indicator such as a monotonically increasing timestamp, a row version column, or a CDC log. In the source, capture changes using CDC tools (e.g., Debezium) or by querying rows where the change column is greater than the last load timestamp. In the pipeline, stage the incremental data into a temporary area, then apply transformations. Use idempotent operations (e.g., upserts keyed on a primary key) to merge into the target. Partition the target table by the change date to enable efficient incremental writes and to keep historical data separate. Finally, update the checkpoint (last load timestamp) and schedule the job. Complexity is O(n) for n new rows, and the approach scales horizontally by partitioning and parallel processing.