Overload, Retry Storm, and Metastable Failure

Overload, Retry Storm, and Metastable Failure A lifecycle diagram generated by Archify. Service health Overload spiral Mitigations Terminal exits Healthy · load within capacity · Service health · normal 01 Healthy load within capacity normal Near Overload · throughput at capacity · Service health · risk 02 Near Overload throughput at capacity risk Queueing · response time rises · Overload spiral · head-of-line Queueing response time rises head-of-line Client Timeouts · requests abandoned · Overload spiral · retries begin Client Timeouts requests abandoned retries begin Retry Storm · requests multiply · Overload spiral · feedback loop Retry Storm requests multiply feedback loop Mitigation On · backoff, breaker, shedding · Mitigations · levers Mitigation On backoff, breaker, shedding levers Recovered · service back to normal · Terminal exits · terminal Recovered service back to normal terminal Metastable Failure · persists after load drops · Terminal exits · needs reset Metastable Failure persists after load drops needs reset load grows near capacity delay exceeds timeout clients retry more load levers applied queue drains no reset reboot / reset Legend initial state active decision success failure / exit final state