Task Failure, Preemption, and Retry in a Batch Job Diagram

Preemption Dominates Hardware Failure

In practice, preemption — not hardware faults — is the most common cause of task loss in a shared cluster
A scheduler may reclaim spot instances or make a node yield to a job with a higher priority
Non-urgent batch jobs are a natural fit for cheap spot instances: they can simply be retried later

Three Intermediate Data Strategies

MapReduce writes every intermediate result to the distributed filesystem and waits for the write: good fault tolerance at a heavy I/O price
Spark keeps intermediate data in memory, spilling to local disk, and writes only the final result to DFS; lost partitions are recomputed from lineage
Flink checkpoints task state periodically, snapshotting it to durable storage instead of materializing every intermediate record

Why Retry at Task Level Is Enough

Tasks read immutable input and have no side effects, so re-running one is always safe
Only the failed task is rescheduled on another machine — the whole job never restarts
Each attempt rebuilds its output from scratch, so retries can never duplicate results

All-or-Nothing Output

When an attempt fails, its partial output is deleted so no half-written data survives
Job output is committed only after every task has finished successfully
If the job fails overall, no output is ever visible to readers