Rebalancing modes
Rebalancing: who moves the data?
Fully automatic
Propose, human commits
Manual
PLUS
no ops toil
scales in minutes
no surprises
admin approves
full control
plan for big events
MINUS
may swamp network
splits lag writes
slower in crisis
needs an on-call human
human toil
slow in crisis
FITS
calm clusters
safest default
before Black Friday
auto rebalance + auto failure detection = cascading failure
a slow node looks dead · the cluster piles on more load