Mixed Reducer Stream: Profile-First or Buffer

Mixed Reducer Stream: Profile-First or Buffer A data-flow diagram generated by Archify. 01 / Input 02 / Shuffle 03 / Order 04 / Reduce Mixed Input Stream · mixed clicks, profiles · 01 / Input Mixed Input Stream mixed clicks, profiles Partition by user_id · same reducer per user · 02 / Shuffle Partition by user_id same reducer per user Secondary Sort · profile first, then ts · 03 / Order · the fix Secondary Sort profile first, then ts the fix Click Buffer · clicks wait here · 03 / Order Click Buffer clicks wait here Reducer Join · emits (url, dob) · 04 / Reduce Reducer Join emits (url, dob) interleaved records interleaved records sort: user, type, ts ordered group per user unsorted: buffer clicks unordered group per user profile first, stream clicks profile arrives first flush buffered clicks flush on profile arrival Legend primary data async batch