Publish/Subscribe Topologies — Load Balancing, Fan-out, and Log Partitions Diagram

Load Balancing Versus Fan-out

Load balancing: each message goes to exactly one consumer, so consumers share expensive work; add consumers to parallelize
Fan-out: each message goes to every subscriber, so independent consumers each get the full broadcast — like several batch jobs reading one file
Both patterns combine: in Kafka, each message goes to one consumer per group (balancing within the group) and to every group (fan-out across groups)

Log Partitions and Offsets

Producers append to a partition; the broker assigns a monotonic offset to every message
Messages in one partition are totally ordered; there is no ordering guarantee across partitions
A consumer group assigns whole partitions to nodes, so at most one node per partition consumes — parallelism is capped by partition count
Route events that must stay ordered (one user's events) to the same partition by key

Acknowledgments and Consumer Offsets

JMS style: the consumer acks each message; the broker redelivers unacknowledged ones — redelivery plus load balancing reorders messages
Log style: the consumer checkpoints its offset periodically and resumes there after a failure, so already-processed messages may repeat
Slow consumers only fall behind on their own offset; other consumers are unaffected, and old segments age out into a bounded disk buffer