Archivedistributed-systems
Failure Detectors Cannot Be Perfect
In an asynchronous network you cannot distinguish a slow node from a dead one. Every detector picks a trade-off.
- Published
- Reading time
- 1 min read
A perfect failure detector would tell you exactly which nodes have crashed. In an asynchronous system with unreliable message delays, that is impossible — a node that has not responded may be slow, partitioned, or dead.
The practical spectrum
- Timeout-based: simple; tune timeout vs false positives. Most systems use adaptive timeouts based on observed RTT.
- Phi accrual (SWIM): outputs a suspicion level from the distribution of inter-arrival times, so applications choose when to act.
- Heartbeat with leases: the absence of a heartbeat is only meaningful after a lease expires.
Consequences
Every false positive triggers unnecessary failover and data movement; every false negative delays recovery. Systems that conflate "unreachable" with "dead" tend to cascade during network incidents.
Takeaway: model failure detection as a suspicion signal with a confidence level. Make failover idempotent, and expect it to fire on healthy nodes during congestion.