HomeInterview QuestionsGiven the services mentioned, here's a scenario: W…

Given the services mentioned, here's a scenario: We have data in either an RDS instance or DynamoDB. We need to load it into Snowflake at regular intervals in an incremental manner. How would you do this?

🟡 Medium System Design Mid level
1Times asked
Jun 2026Last seen
Jun 2026First seen

💡 Model Answer

For incremental loads from RDS or DynamoDB to Snowflake, I would build a hybrid CDC pipeline:

  1. Change Capture – For RDS, enable binlog and use DMS or a custom MySQL binlog reader. For DynamoDB, use DynamoDB Streams. Both emit change events with operation type and timestamp.
  2. Event Bus – Push events to Kinesis Data Streams. Create separate shards for RDS and DynamoDB to isolate traffic.
  3. Batching & Transformation – A Lambda or Glue job reads from Kinesis, aggregates events into micro‑batches (e.g., 5‑minute windows), converts them to Parquet, and writes to an S3 landing bucket.
  4. Snowflake Ingestion – Snowpipe monitors the landing bucket. Each file is loaded into a staging table. Snowflake streams on the staging table capture new rows.
  5. Merge Logic – A scheduled Snowflake task runs a MERGE statement that applies INSERT, UPDATE, DELETE based on the operation flag and a primary key. This ensures incremental consistency.
  6. Scheduling – Use AWS EventBridge to trigger the Lambda/Glue job at fixed intervals (e.g., every 5 minutes). Snowflake tasks run every 10 minutes to keep the target table fresh.
  7. Observability – CloudWatch dashboards for Kinesis and Lambda, Snowflake query history for load success, and alerts for failures.

This architecture supports both relational and NoSQL sources, scales with data volume, and keeps Snowflake up‑to‑date with minimal latency.

This answer was generated by AI for study purposes. Use it as a starting point — personalize it with your own experience.

🎤 Get questions like this answered in real-time

Assisting AI listens to your interview, captures questions live, and gives you instant AI-powered answers — invisible to screen sharing.

Get Assisting AI — Starts at ₹500