Let's say I have to design a pipeline. How do I decide whether to use incremental load or full load?

When deciding between incremental and full load, consider data volume, change frequency, source capabilities, latency requirements, and resource constraints. If the source supports change data capture (CDC) or provides a timestamp/sequence column, incremental load is usually preferable because it processes only new or updated rows, reducing load time and storage usage. It is ideal for large datasets or near‑real‑time pipelines. If the source is small, changes are infrequent, or CDC is not available, a full load may be simpler and less error‑prone. Full refreshes also guarantee consistency when the source schema changes. In practice, many teams adopt a hybrid approach: perform a full load initially to seed the target, then use incremental loads for subsequent runs. Complexity-wise, a full load is O(n) where n is the total rows, while incremental is O(k) where k is the number of changed rows, making incremental more efficient when k << n.