Archivereliability
The Thundering Herd Is a Synchronization Bug
Aligned timers, synchronized reconnects, and simultaneous retries all convert a small event into a spike.
- Published
- Reading time
- 1 min read
A thundering herd is many clients reacting to the same event at the same instant: a cache flush, a deploy, a DNS TTL expiry, a leader election.
Common triggers
- All instances restart with the same cron minute.
- Clients reconnect after a load balancer drops connections, with no backoff jitter.
- Cache entries written together expire together.
- Health checks fail simultaneously and all instances report unhealthy.
The fixes
- Jitter every scheduled action: cron offsets, retry delays, TTLs, reconnect timers.
- Spread deploys with rolling updates and readiness gates.
- Backoff with randomization on reconnect and retry.
- Rate-limit recovery: allow the dependency to ramp, not to receive full traffic immediately.
The deeper issue
Synchronization is often implicit — identical configuration applied to identical instances. If every replica has the same timer, you have built a herd by construction.
Takeaway: assume every periodic action will align unless you randomize it. Design recovery paths to ramp, not to snap.