Retries Turn a Blip Into an Outage

A retry is a bet that the failure was transient. When many clients retry at once, the bet becomes a traffic multiplier against an already-degraded dependency.

The amplification loop

1. A dependency slows down.
2. Clients time out and retry immediately.
3. Load doubles or triples; the dependency slows further.
4. Timeouts fire more; retries multiply. Recovery becomes impossible until clients are killed.

What actually helps

• Exponential backoff with full jitter: sleep = random(0, min(cap, base * 2^attempt)). Jitter is the important part.
• Retry budgets: cap retries as a fraction of total requests (e.g. 10%), so retries cannot dominate.
• Circuit breakers: stop sending when the failure rate crosses a threshold.
• Idempotency: retries must be safe for non-idempotent operations.
• Hedged requests with care: they help tail latency but add load.

Takeaway: retries are load. Budget them globally, randomize them, and make the operation idempotent or do not retry it.