Mini Shuffle, End to End

Mini Shuffle, End to End A data-flow diagram generated by Archify. 01 / Map 02 / Shuffle 03 / Reduce 04 / Output Immutable Splits · one per mapper · 01 / Map Immutable Splits one per mapper Mappers · partition, sort, combine · 02 / Shuffle · same hash Mappers partition, sort, combine same hash Partition Fetch · one R per reducer · 03 / Reduce Partition Fetch one R per reducer Lineage Rerun · only the lost mapper · 03 / Reduce Lineage Rerun only the lost mapper Grouped Output · same key together · 04 / Output Grouped Output same key together read splits immutable splits sorted mPrR files sorted mPrR files merged partitions grouped partitions checksum fails: rerun mapper checksum fails Legend primary data async batch data store