Write a simple code skeleton for an incremental data model.

Question

Assisting AI · Accepted Answer

Here is a Python‑style skeleton that demonstrates a typical incremental load:

```python
# 1. Define source and target tables
source_table = 'staging_raw'
target_table = 'dim_customer'

# 2. Identify the key and timestamp columns
key_col = 'customer_id'
timestamp_col = 'last_modified'

# 3. Fetch the most recent timestamp from the target
last_ts = get_max_timestamp(target_table, timestamp_col)

# 4. Pull only new or updated rows from the source
new_rows = query(
    f"SELECT * FROM {source_table} WHERE {timestamp_col} > '{last_ts}'"
)

# 5. Merge into the target (upsert logic)
for row in new_rows:
    upsert(target_table, key_col, row)

# 6. Commit transaction
commit()
```

**Explanation**: The script first determines the latest timestamp in the target dimension. It then queries the source for rows newer than that timestamp, ensuring only incremental changes are processed. The `upsert` function handles inserts or updates based on the primary key. Complexity is O(n) for n new rows, and the approach scales well for large datasets when combined with batch processing or parallelism.

Write a simple code skeleton for an incremental data model.

💡 Model Answer

🎤 Get questions like this answered in real-time