Archiveperformance
Tail Latency Is a Fan-Out Problem
If one request touches 100 services, the p99 of each becomes the p50 of the whole.
- Published
- Reading time
- 1 min read
Averages hide the experiences users remember. Worse, fan-out turns modest per-service tails into a system-wide median.
The math
If a request fans out to n independent backends, each with probability p of being slow, the chance at least one is slow is:
P(slow request) = 1 - (1 - p)^n
With p = 1% and n = 100, 63% of requests hit at least one slow backend. The p99 of the leaf becomes the typical experience of the root.
What to do
- Hedge slow requests: issue a duplicate after p95 and take the first response — at the cost of extra load.
- Reduce fan-out: batch, cache, or precompute.
- Isolate: separate pools per dependency so one slow service does not starve others.
- Shed: return partial results when a dependency is slow.
- Measure per-hop, not just end-to-end, so you can see where the tail comes from.
Takeaway: optimize the tail of the leaves or reduce the number of leaves. The mean of the root is a fiction.