Exactly-once: inside and outside the framework
Exactly-Once in Streams — Inside and Outside
No “end” to wait for: checkpoint inside, commit or idempotence outside.
INSIDE THE FRAMEWORK — same trick as batch
MICROBATCH (Spark)
cut the stream into ~1s blocks
each block runs like a mini-batch
failed block restarts, partial dropped
CHECKPOINT (Flink)
rolling snapshots of state
barrier-triggered, no fixed size
restart from latest, drop the gap
OUTSIDE EFFECTS — the framework cannot un-send
ATOMIC COMMIT
state + messages + offset
persist all-or-nothing, only if processing succeeded
kept inside one framework
IDEMPOTENT SINK
store Kafka offset with the write
repeats become no-ops
needs same-order replay + fencing
exactly-once = effectively-once:
visible effect as if processed once — even when retried.
state recovery: remote store, replicated local, or rebuild from inputs.