How would you configure incremental loading for five tables while the remaining five tables use full refresh?
💡 Model Answer
Incremental loading captures only new or changed rows, reducing load time and resource usage. First, determine a change indicator in each incremental table, such as a timestamp column (e.g., last_updated) or a CDC flag. In the ETL tool, create a parameter that stores the last successful run timestamp. During extraction, query the source with a WHERE clause that filters rows where the change indicator is greater than the stored timestamp. In the transformation step, you may need to upsert (merge) the rows into the target: insert new rows and update existing ones based on a primary key. After the load, update the stored timestamp to the maximum value processed. For the five full‑refresh tables, use the approach from question 1. You can manage both types in a single DAG by branching logic or by separate tasks that share the same schedule. This ensures that incremental tables stay current with minimal overhead while full‑refresh tables are refreshed entirely each run.
This answer was generated by AI for study purposes. Use it as a starting point — personalize it with your own experience.
🎤 Get questions like this answered in real-time
Assisting AI listens to your interview, captures questions live, and gives you instant AI-powered answers — invisible to screen sharing.
Get Assisting AI — Starts at ₹500