Your payments producer is hitting a hot shard on Kinesis Data Streams during peak hours, causing iterator age spikes and throttling. Within a 15‑minute mitigation window, what steps would you take to stabilize throughput (e.g., split shards, use temporary randomized sub‑keys) and what is your long‑term keying strategy to prevent recurrence while keeping ordering guarantees where needed?

In the short term, I would first identify the hot shard by looking at the shard metrics in CloudWatch. Then I would split the shard using the Kinesis API (SplitShard) to create two shards, each handling half of the hash key range. If the producer can’t immediately adjust its hash key, I would temporarily add a random prefix to the partition key (e.g., paymentId#random) to spread the traffic across the new shards. I would also increase the iterator age limit and enable enhanced fan‑out if the consumer side is the bottleneck. For the long‑term strategy, I would redesign the partition key to be a composite of the merchant ID and a deterministic hash of the payment ID, ensuring even distribution while preserving ordering per merchant. I would also implement a key‑sharding function that maps each merchant to a specific shard range and use a consistent hashing algorithm to avoid hotspots. Finally, I would enable Kinesis Data Streams’ automatic shard scaling or use Kinesis Data Streams with Lambda to dynamically adjust shard count based on traffic patterns, while keeping ordering guarantees by grouping related events under the same partition key.