Shard Move Lifecycle — Copy, Catch Up, Cutover, Abort Diagram

The Move, Step by Step

Rebalancing starts when a node joins or leaves; the shard map is the only authority on who owns what
Bulk copy streams the data while writes continue — so a catch-up phase closes the gap
Cutover is one atomic update of the shard map, not a slow migration of requests
After cutover the old owner stops accepting writes for that shard and frees its copy

The Cutover Window

Between the map update and every client hearing about it, requests to the old owner are still in flight
The safe answers: forward them to the new owner, or reject them so the client retries
A cached routing table in a client can strand writes on the wrong node if nothing invalidates it
Coordination services notify subscribers on change precisely to shrink this window

Aborting a Move

A move can fail halfway: network trouble, disk full, or the target node dying
If the shard map was never updated, the old owner is still authoritative — aborting loses nothing
Data must never be deleted from the source until the new owner is confirmed
Automatic rebalancing multiplies every one of these risks; a human in the loop is often the safer design