Revisiting Aggregation for Data Intensive Applications: A Performance Study

TL;DR

This paper provides an in-depth performance analysis of aggregation algorithms in Big Data platforms, highlighting implementation challenges and proposing cost models to guide algorithm selection based on system parameters.

Contribution

It offers a comprehensive performance study of aggregation algorithms, discusses implementation details, and introduces cost models for better algorithm choice in Big Data systems.

Findings

Implementation of aggregation algorithms is complex and influenced by multiple factors.

Memory, spilling strategy, and I/O significantly affect performance.

Cost models can predict the best algorithm based on input parameters.

Abstract

Aggregation has been an important operation since the early days of relational databases. Today's Big Data applications bring further challenges when processing aggregation queries, demanding adaptive aggregation algorithms that can process large volumes of data relative to a potentially limited memory budget (especially in multiuser settings). Despite its importance, the design and evaluation of aggregation algorithms has not received the same attention that other basic operators, such as joins, have received in the literature. As a result, when considering which aggregation algorithm(s) to implement in a new parallel Big Data processing platform (AsterixDB), we faced a lack of "off the shelf" answers that we could simply read about and then implement based on prior performance studies. In this paper we revisit the engineering of efficient local aggregation algorithms for use in Big Data platforms. We discuss the salient implementation details of several candidate algorithms and present an in-depth experimental performance study to guide future Big Data engine developers. We show that the efficient implementation of the aggregation operator for a Big Data platform is non-trivial and that many factors, including memory usage, spilling strategy, and I/O and CPU cost, should be considered. Further, we introduce precise cost models that can help in choosing an appropriate algorithm based on input parameters including memory budget, grouping key cardinality, and data skew.