State and Streams Are Two Sides of the Same Coin (Figure 12-7)
Chapter 12 — the changelog and the table are the same data, viewed at different points in time
event stream / changelog
offset 0 · (mew, 12)
offset 1 · (purr, 3)
offset 2 · (mew, 17)
offset 3 · (scratch, 5)
offset 4 · (mew, 21)
offset 5 · …
«append-only · immutable · replayable»
application state
key · latest value
mew · 21
purr · 3
scratch · 5
reads: O(1) lookup by key
«mutable · read-optimized · keyed lookup»
integrate over time
(apply events)
differentiate by time
(change data capture)
“The log contains all the information there is; the database is just the end of the log.”
— after J. Gray & A. Reuter, Transaction Processing: Concepts and Techniques (1992)
log compaction — keep only the latest value per key
before — full history
offset 0 · (mew, 12)
offset 1 · (purr, 3)
offset 2 · (mew, 17)
offset 3 · (scratch, 5)
offset 4 · (mew, 21)
offset 5 · (purr, ⊥)
compact
keep latest per key
after — compaction
offset 0 · (mew, 21)
offset 1 · (scratch, 5)
offset 2 · (purr, ⊥) tombstone
why it matters
compacted log ≈ full copy of the database;
new derived systems can replay from offset 0
tombstones age out after a retention period