Many of us handle incremental data loading. What was the daily data volume of yours? Did you use CDC watermarking, merge, append, or any other approach?

Incremental data loading is the process of capturing only new or changed records since the last load, which reduces processing time and resource usage. Common approaches include Change Data Capture (CDC), which records changes in a source system’s transaction log; watermarking, which uses a monotonically increasing timestamp or sequence number to identify new rows; merge operations, which upsert records into a target table based on a key; and append-only loads, which simply add new rows without deduplication. CDC is ideal when the source supports log-based capture, providing near real‑time updates. Watermarking works well for batch pipelines where a reliable timestamp exists. Merge is useful when you need to maintain a consistent target state and handle updates or deletes. Append is simplest but requires downstream deduplication if duplicates can occur. The choice depends on data volume, latency requirements, source capabilities, and target schema. For example, in a daily ETL for a 10‑GB log table, I used CDC via Debezium to stream changes to Kafka, then a Spark Structured Streaming job performed a merge into a Delta Lake table, ensuring idempotent writes and efficient incremental processing.