From Partition Key to Node — Hash Sharding Paths

From Partition Key to Node — Hash Sharding Paths A data-flow diagram generated by Archify. 01 / Partition Key 02 / Hash 03 / Shard Mapping 04 / Rebalancing Partition Key · key or key prefix · 01 / Partition Key · skewed input ok Partition Key key or key prefix skewed input ok Clustered Keys · nearby keys cluster · 01 / Partition Key · hot shard risk Clustered Keys nearby keys cluster hot shard risk Hash Function · Murmur3 or MD5 · 02 / Hash · not cryptographic Hash Function Murmur3 or MD5 not cryptographic Uniform Spread · ordering is lost · 02 / Hash · no key range scans Uniform Spread ordering is lost no key range scans Fixed Shards · hash(key) % 1000 · 03 / Shard Mapping · count fixed up front Fixed Shards hash(key) % 1000 count fixed up front Hash Ranges · ranges of hash values · 03 / Shard Mapping · count adapts to data Hash Ranges ranges of hash values count adapts to data Move Whole Shards · shard to node map · 04 / Rebalancing · cheap rebalancing Move Whole Shards shard to node map cheap rebalancing Split and Merge · rewrite to new files · 04 / Rebalancing · expensive operation Split and Merge rewrite to new files expensive operation hash the partition key mapping map hash to a shard fixed count node joins: move shards rebalancing nearby keys scatter apart skew relief assign ranges of hashes adaptive count shard too big: split it rebalancing Legend primary data data store data flow