Batch Processing Models and Their Building Blocks
Chapter 11 conceptual map — processing models · storage · orchestration, plus the shuffle between stages
1 · processing models
2 · storage
3 · orchestration
ProcessingModel
«abstract · bounded input → derived output»
input : immutable dataset, read once
output : derived dataset · terminates
UnixToolPipeline
«sort-based · single machine»
tools piped over stdin/stdout
sort merges records by key
DataflowEngine
«whole workflow = one job · operators fused»
pipelined · intermediate data stays in memory
SQL/DataFrame APIs compile to this layer
examples: Spark / Flink style engines
MapReduce
«map, implicit sort, reduce»
file-based I/O between stages
per-task retry on failure
SqlQueryEngine
«SQL → physical plan»
cost-based optimizer
join order is rearranged
DataFrameApi
«relational ops as methods»
Pandas-style API running on
a distributed engine
Shuffle
«shared helper · repartition»
hash(key) → one sorted file
per reducer · merge-sorted
Storage
«pluggable read/write protocol»
S3 API is the de-facto standard
DistributedFS
«HDFS-style distributed FS»
large blocks (128 MB) on data nodes
metadata service · replication
or erasure coding · locality hints
ObjectStore
«S3-style bucket + key»
immutable objects · no locks
no atomic rename of objects
storage and compute separated
JobScheduler
«one job»
task executors + resource mgr
FIFO / DRF / priority / quotas
WorkflowScheduler
«DAG of jobs»
waits for upstream outputs
Airflow-style orchestration
Orchestrator
«abstract · runs batch jobs»
decides where and when jobs run
«compile to»
«compile to»
reads input / writes output
through pluggable S3 API
runs under
submits jobs to
always uses
uses only where needed
Legend
inheritance
association
«compile to»