A Timeout Happened — Now What? Retry, Reconcile, Guard, or Fail Diagram

Step One: Accept the Ambiguity

A timeout never tells you the request failed — it only tells you no reply arrived inside your budget
The request may be queued in a switch, the node may be paused in a GC cycle, or the reply may be the only thing that was lost
So the first decision is not 'did it work?' but 'can I safely act again?'

Three Safe Recoveries

Idempotency keys make retries free: the server collapses repeats of the same key into one effect
Reading the state back answers the question directly — ask the service what happened instead of guessing
Fencing tokens protect writes from stale owners: storage rejects any write carrying an older token

When to Give Up

If none of the three apply, failing loudly is better than guessing: surface the error and let a human or a workflow decide
Retries need a budget — unlimited retries turn a slow node into a self-inflicted denial of service
Back off with jitter so retried clients do not synchronize into a thundering herd