Two Sorted Streams, One Streaming Merge

Two Sorted Streams, One Streaming Merge A data-flow diagram generated by Archify. 01 / Inputs 02 / Sort 03 / Merge 04 / Output Click Events · user_id, url, ts · 01 / Inputs · bigger than RAM Click Events user_id, url, ts bigger than RAM Profiles · user_id, dob, unique · 01 / Inputs · one per user Profiles user_id, dob, unique one per user Sort Clicks · by user_id, spill · 02 / Sort Sort Clicks by user_id, spill Sort Profiles · by user_id, spill · 02 / Sort Sort Profiles by user_id, spill Streaming Merge · two cursors, one profile · 03 / Merge · O(1) memory Streaming Merge two cursors, one profile O(1) memory Joined Visits · (url, dob) per visit · 04 / Output Joined Visits (url, dob) per visit Missing-Profile Bin · left-join NULL + counter · 04 / Output Missing-Profile Bin left-join NULL + counter sort clicks user_id, url, ts sort profiles user_id, dob, unique clicks by user_id ordered by user_id profiles by user_id ordered by user_id (url, dob) per visit one pair per visit missing counted counted, never silent Legend primary data async batch data store