Skip to content
hashkey.bond

Archiveperformance

Cache Stampedes and the Thundering Herd

When a hot key expires, every request misses at once. Single-flight and jittered TTLs fix it.

Published
Reading time
1 min read

A popular cache entry expires. Hundreds of concurrent requests miss, all query the database for the same row, and the database — which was comfortably sized for cache hits — falls over.

Mitigations

  1. Single-flight / request coalescing: the first miss computes; others wait for the same result. Go's singleflight, or a per-key lock with a short wait.
  2. Probabilistic early expiration: refresh before expiry with a probability that rises as the TTL shrinks (XFetch). Spreads refreshes over time.
  3. TTL jitter: add ±10% randomness so keys do not expire in lockstep.
  4. Stale-while-revalidate: serve stale data while one worker refreshes.
  5. Negative caching: cache "not found" briefly to protect against repeated misses.

The subtlety

The lock must not become the new bottleneck. A per-key lock is fine; a global lock serializes unrelated keys. And if the compute path fails, you need a fallback rather than a queue of waiting requests.

Takeaway: every hot key needs a refresh strategy that assumes concurrency. Expiry is a scheduled traffic spike unless you design it otherwise.