Sort-Merge Join, End to End

Sort-Merge Join, End to End A data-flow diagram generated by Archify. 01 / Inputs 02 / Sort 03 / Merge 04 / Serve Click Events · user_id, url, ts · 01 / Inputs · bigger than RAM Click Events user_id, url, ts bigger than RAM Profiles · user_id, dob, unique · 01 / Inputs · declare uniqueness Profiles user_id, dob, unique declare uniqueness Sort Clicks · by user_id, spill · 02 / Sort Sort Clicks by user_id, spill Sort Profiles · by user_id, spill · 02 / Sort Sort Profiles by user_id, spill Version Rule · click joins live-at-ts · 03 / Merge · history rule Version Rule click joins live-at-ts history rule Streaming Merge · two cursors, one profile · 03 / Merge · O(1) memory Streaming Merge two cursors, one profile O(1) memory Joined Visits · (url, dob) per visit · 04 / Serve Joined Visits (url, dob) per visit Missing Bin · left-join NULL + counter · 04 / Serve Missing Bin left-join NULL + counter sort clicks user_id, url, ts sort profiles user_id, dob, maybe history clicks by user_id ordered by user_id profiles by user_id ordered by user_id live at ts live version (url, dob) per visit one pair per visit missing counted counted, never silent Legend primary data async batch data store