Sort-Merge Join and Group-By in Two Batch Jobs

Sort-Merge Join and Group-By in Two Batch Jobs A data-flow diagram generated by Archify. 01 / Sharded Inputs 02 / Map Phase 03 / Shuffle + Sort 04 / Join Reduce 05 / Group-By Job Clickstream · fact table · 01 / Sharded Inputs · user_id, ts, URL Clickstream fact table user_id, ts, URL User Profiles · dimension table · 01 / Sharded Inputs · user_id, DOB User Profiles dimension table user_id, DOB Event Mapper · key: user_id · 02 / Map Phase · URL + timestamp Event Mapper key: user_id URL + timestamp Profile Mapper · key: user_id · 02 / Map Phase · date of birth Profile Mapper key: user_id date of birth User-ID Shuffle · group by user_id · 03 / Shuffle + Sort · DOB first, then ts User-ID Shuffle group by user_id DOB first, then ts Join Reducer · sort-merge join · 04 / Join Reduce · one user in RAM Join Reducer sort-merge join one user in RAM URL Shuffle · group by URL · 05 / Group-By Job · job 2 starts here URL Shuffle group by URL job 2 starts here Age Counter · per-URL counters · 05 / Group-By Job · increment per view Age Counter per-URL counters increment per view Page Age Stats · age distribution · 05 / Group-By Job · output dataset Page Age Stats age distribution output dataset clickstream profile records mapped events mapped profiles one user's data secondary sort joined pairs URL + DOB grouped views age counts Legend primary data async batch data store data flow