Load Shedding: Serve Fewer Requests, Correctly

When demand exceeds capacity, the system must drop something. The only question is whether the drop happens at the edge, deliberately, or in the middle, chaotically.

Strategies

• Admission control: reject when concurrency or queue depth exceeds a limit, returning 429 with Retry-After.
• Priority shedding: protect checkout and health checks; drop analytics and prefetch first.
• Deadline-based shedding: if a request cannot finish within its deadline, reject it now rather than after consuming resources.
• Per-tenant fairness: token buckets per customer prevent one noisy tenant from starving others.

Why early is better

A request that will fail anyway still consumes a connection, CPU, and queue slot. Dropping it at ingress frees capacity for work that can succeed.

Measure

Track shed rate, shed reason, and the latency of admitted requests. If admitted latency stays flat while shedding rises, the mechanism is working; if latency climbs, you are shedding too late.

Takeaway: an explicit shedding policy is a feature. Without one, the system sheds by timing out, which is the most expensive way to say no.