Where the Time Goes — MapReduce vs a Dataflow Engine
one shared time axis · green bars are compute tasks · grey bars are durable writes to the filesystem
compute task
durable write to DFS / object store
stage barrier
time →
not to scale
① MapReduce — every stage finishes by writing files to the distributed filesystem
map 1
read shard → map()
map 2
read shard → map()
map 3
read shard → map()
stage barrier
DFS write
write intermediate to DFS (replicated, durable)
stage barrier
reduce 1
reduce 1
reduce 2
reduce 2
reduce 3
reduce 3
stage barrier
output
write output
file-based I/O between every stage · no pipelining · new JVM per task · sort happens between every map and reduce
② Dataflow engine (Spark / Flink) — one job, one DAG: operators start as soon as their input is ready
map + filter
map+filter fused into one task
map 2
read shard → map()
reduce 1
reduce 1 · starts as soon as input is ready
reduce 2
reduce 2
intermediate
intermediate state: memory / local disk
output
final output → DFS / object store
sort only where needed · whole DAG visible to the scheduler → data locality
reuse worker processes · lineage recompute / checkpoints instead of DFS writes