Hash Aggregation Under a Memory Cap

Hash Aggregation Under a Memory Cap A data-flow diagram generated by Archify. 01 / Source 02 / Extract 03 / Partition 04 / Aggregate 05 / Report access.log · immutable input · 01 / Source · teaching log access.log immutable input teaching log Extract URL · field 7 per line · 02 / Extract Extract URL field 7 per line Hash Partitioner · hash(url) % P · 03 / Partition · bounded RAM Hash Partitioner hash(url) % P bounded RAM Partial Aggregates · counters in RAM · 04 / Aggregate Partial Aggregates counters in RAM Spill Files · one run per partition · 04 / Aggregate Spill Files one run per partition Top-K Rank · sum, sort, head · 05 / Report Top-K Rank sum, sort, head read log lines immutable log lines URL stream one URL per line hot partitions aggregate in RAM hot keys stay resident cold partitions spill to disk cold keys leave RAM partial counts exact partial counts merged spill counts spilled counts re-merged Legend primary data async batch data store