Archiveperformance
Never Average Percentiles
Percentiles are not additive. Aggregating p99s across shards or minutes produces a number that means nothing.
- Published
- Reading time
- 1 min read
The p99 of a union is not the average of the p99s. It is not the max either. If you average per-minute p99 values, a single minute with a huge tail can be diluted by 59 quiet minutes.
What to do instead
- Merge histograms: keep latency histograms (HDR or exponential buckets) and merge counts before computing quantiles. This is what Prometheus histograms and OpenTelemetry exponential histograms enable.
- Keep raw samples for smaller volumes and compute quantiles over the union.
- Report max and p999 alongside p99 when tails matter.
Why it matters operationally
A dashboard that averages replica p99s can show a healthy 120 ms while one replica is at 4 s and serving 10% of traffic. The average structure hides the exact node you need to fix.
Takeaway: aggregate distributions, not summary statistics. If your metrics backend only stores percentiles, you cannot correctly combine them.