When pulling JSON payload from an API, how do you ensure you don't retrieve the same data again? For example, if you pull data today, tomorrow you only want data that has changed on the source side.

To avoid re‑pulling unchanged data, I would use a combination of metadata and incremental fetching strategies. First, the API should expose a 'last_modified' timestamp or a monotonically increasing version ID for each record. The client stores the maximum timestamp/version seen. On subsequent pulls, it sends this value as a query parameter (e.g., ?since=2024-05-23T12:00:00Z). The server then returns only records newer than that value. If the API supports ETag or Last‑Modified headers, the client can include If‑None‑Match or If‑Modified-Since; a 304 response indicates no changes. For large datasets, a delta endpoint or CDC stream (e.g., Kafka, AWS Kinesis) can push only changed rows. Additionally, idempotent operations and de‑duplication logic on the client side ensure that even if duplicates slip through, they are ignored. This approach minimizes data transfer, reduces processing load, and keeps the local store in sync with the source.