Hot Keys: The Complete Guide Across All Systems

Every scalable data system partitions work by a key: Kafka by message key, Spark by shuffle key, Flink by keyBy, DynamoDB/Cassandra/Bigtable by partition key, Redis Cluster by hash slot. Partitioning scales only if load spreads evenly across keys. A hot key is a key whose traffic or data volume is so large that the single partition, task, node or shard owning it becomes the bottleneck, while the rest of the cluster sits idle.

In system design interviews, raising hot keys before the interviewer does is one of the clearest senior signals. This module gives you the vocabulary, the per-system behaviour and a playbook.

flowchart LR
    subgraph Even["Healthy: load spread evenly"]
        direction TB
        K1[keys] --> P1["P0 ▮▮▮"]
        K1 --> P2["P1 ▮▮▮"]
        K1 --> P3["P2 ▮▮▮"]
    end
    subgraph Hot["Hot key: one partition does all the work"]
        direction TB
        K2[keys] --> Q1["P0 ▮▮▮▮▮▮▮▮▮▮▮▮ celebrity_id"]
        K2 --> Q2["P1 ▮"]
        K2 --> Q3["P2 ▮"]
    end
    Even ~~~ Hot
    style Q1 fill:#fee2e2,stroke:#ef4444,color:#7f1d1d

1. Where hot keys come from

Two different problems hide under one name:

• Hot in traffic: too many requests per second for one key (read or write throughput).
• Hot in size: too much data for one key (a 200 GB group in a shuffle, an unbounded state entry).

The fixes differ, so say which one you mean.

2. How each system suffers

Kafka (and Kinesis, Pub/Sub ordering keys)

• The producer’s partitioner sends all messages with the same key to one partition. A hot key means one partition receives most of the traffic.
• Consequences: one broker’s disk and network saturate; the one consumer assigned that partition lags (consumer parallelism is capped at one per partition per group); end-to-end latency grows for all keys sharing the partition.
• Detection: per-partition bytes-in and messages-in, per-partition consumer lag (max vs median).
• Mitigations:
  • Key by a finer-grained key (user_id instead of merchant_id) if per-merchant ordering isn’t required.
  • Key salting: merchant_42#3 with a salt 0..N-1, then re-aggregate downstream. Ordering is only preserved per salted key.
  • For keyless events, use the sticky/round-robin partitioner.
  • More partitions don’t help a single hot key. Say this explicitly.

Spark (batch shuffles, joins, aggregations)

• A hot key’s rows all land in one shuffle partition, so one task runs for an hour while 1,999 finish in a minute; that task spills, may OOM, and can cause fetch failures upstream.
• Mitigations: filter junk keys, broadcast the small side, AQE skew join, salting, two-phase aggregation. See shuffle, spill & salting.

Flink / Spark Structured Streaming / Kafka Streams (keyed state)

• keyBy(merchant) sends every event of a merchant to one parallel subtask: CPU hot spot plus state hot spot (RocksDB state for that key grows large; checkpoints slow down).
• Backpressure from the hot subtask propagates upstream and slows the whole job, including cold keys.
• Mitigations:
  • Local (pre-)aggregation: aggregate per key within each upstream subtask for a short interval or count, then send partial results to keyBy. This is a two-phase aggregation, the streaming version of salting. Flink SQL has built-in mini-batch and local-global aggregation options for exactly this.
  • Salted keys + second-level aggregation for decomposable aggregates.
  • State TTL to bound per-key state; split large per-key collections.

Databases and key-value stores

Caches (Redis, Memcached, CDN)

• A viral item’s key hits one Redis shard (one hash slot), saturating that node’s CPU or network.
• Cache stampede: when a hot key expires, thousands of requests miss at once and all hit the database.
• Mitigations:
  • Local in-process cache (L1) in front of Redis for the hottest keys, with a short TTL.
  • Replicate hot keys under several names (item:42#0..7) and read a random replica.
  • Request coalescing / single-flight: one request recomputes, the others wait.
  • Early probabilistic refresh (refresh before expiry) and TTL jitter.
  • Push static content to the CDN edge.

APIs and rate limiters

• A single tenant or API key generating most traffic starves others.
• Fixes: per-tenant quotas, token buckets, fair queueing, isolating large tenants on dedicated capacity (“cell” architecture).

3. Detection: measure before you fix

1. Distribution metrics per partition: look at max vs median, not averages. Averages hide hot spots.
2. Top-k keys: a heavy-hitter sketch (Count-Min Sketch + heap, or Space-Saving) on the stream; groupBy(key).count().orderBy(desc) on a sample in batch.
3. Symptoms:
   • Kafka: one partition with lag growing while others are at zero.
   • Spark: max task time ≫ median, one task with huge shuffle read and spill.
   • Flink: one subtask at 100% busy with backpressure upstream.
   • DynamoDB: ThrottledRequests with consumed capacity well below provisioned (the classic hot-partition signature).
   • Redis: one shard’s CPU or network at its limit while others are idle.
4. Alert on skew ratios: e.g. max_partition_lag / median_partition_lag > 10 or the top key’s share of traffic > 5%.

4. The mitigation playbook

flowchart TD
    H[Hot key detected] --> J{Junk key?<br/>NULL, default, bot}
    J -->|yes| F[Filter / route separately / fix upstream]
    J -->|no| T{Hot in traffic<br/>or in size?}
    T -->|read traffic| C[Cache: local L1 + replicas,<br/>request coalescing, CDN]
    T -->|write traffic| W{Need strict per-key order?}
    W -->|no| S[Salt / shard the key,<br/>aggregate downstream]
    W -->|yes| B[Batch writes per key,<br/>dedicated partition or cell,<br/>vertical scale that shard]
    T -->|data size| D{Aggregation or join?}
    D -->|aggregation| P[Two-phase / local pre-aggregation]
    D -->|join| X[Broadcast small side,<br/>AQE skew join, salt hot keys only]

The fundamental trade-off: spreading a key across N places multiplies the cost of reading it (fan-out and merge) and weakens per-key ordering or atomicity. Only spread as much as the load requires, and only for the keys that need it (detect the top k and salt them, not everything).

5. Worked design examples

Interview questions