How do you adjust the watermark if there's a sudden shift in the data?

In a streaming system, the watermark represents the point in event time up to which the system assumes all events have arrived. When a sudden shift in the data occurs—such as a spike in late events or a change in source latency—you need to adapt the watermark strategy to avoid both data loss and excessive lateness. First, monitor the distribution of event timestamps and the lateness histogram in real time. If you notice a persistent tail of late events, increase the allowed lateness or the watermark lag. One common technique is to use a percentile‑based watermark: compute the 95th or 99th percentile of the event timestamps over a sliding window and set the watermark to that value minus a safety margin. This automatically scales with the data distribution. Additionally, you can implement a dynamic watermark that reacts to observed lateness: if the lateness exceeds a threshold for a certain duration, bump the watermark forward; if lateness drops, you can tighten it. Finally, ensure that downstream operators are configured to handle the new watermark—e.g., adjust state TTLs or trigger intervals—to maintain correctness while keeping latency low.