Exactly-once counter: keyed partitions, one owner each, atomic sink

Exactly-once counter: keyed partitions, one owner each, atomic sink An architecture diagram generated by Archify. Click producers · key = user id · Architecture component Click producers key = user id Partition 0 · offsets 0,1,2... · Kafka topic clicks: 3 partitions Partition 0 offsets 0,1,2... Partition 1 · offsets 0,1,2... · Kafka topic clicks: 3 partitions Partition 1 offsets 0,1,2... Partition 2 · offsets 0,1,2... · Kafka topic clicks: 3 partitions Partition 2 offsets 0,1,2... Consumer C1 · owns P0 + P1 · consumer group G: one owner per partition Consumer C1 owns P0 + P1 Consumer C2 · owns P2 · consumer group G: one owner per partition Consumer C2 owns P2 click_counts + checkpoints · one txn per batch · record DB: transactional sink click_counts + checkpoints one txn per batch append keyed append keyed append keyed ordered read ordered read ordered read txn: totals + ckpt txn: totals + ckpt Kafka topic clicks: 3 partitions consumer group G: one owner per partition record DB: transactional sink Legend Backend Database Message bus External