A Distributed Batch Framework Is a Distributed Operating System Diagram

A batch framework has filesystems, job schedulers, and programs that pass data to one another through the filesystem
Storage layer = the filesystem: blocks, metadata, and a pluggable access protocol
Orchestration layer = the kernel: who runs, where, and with how much memory and CPU
Compute layer = the processes: isolated tasks that read an input shard and write an output shard

Blocks are far larger than local ones: 128MB (HDFS) or 4MB (JuiceFS, many object stores) versus 4,096 bytes on ext4
Larger blocks mean less metadata on petabyte-sized datasets and cheaper seeks relative to reads
Blocks are replicated across machines or erasure-coded (Reed–Solomon) for lower storage overhead than full copies
Object stores: bucket + key, objects immutable once written, get/put only — no renames, locks, or real directories
The access protocol is pluggable: the S3 API is implemented by MinIO, R2, B2, Tigris and others, so any of them can back a job

A task can run on any node that holds a replica of its input data
Reading locally saves network bandwidth when the task's code is smaller than the data it reads
Object stores keep storage and compute separate: CPU and memory scale independently of storage, at the cost of network traffic

A job submission names: number of tasks, memory/CPU/disk per task, job id, credentials, parameters, required hardware, and the location of the code
The resource manager keeps the global cluster state, typically in a coordination service such as ZooKeeper or etcd
The workflow scheduler chains jobs into a DAG and submits each job once its dependencies finish (Airflow-style)