Processing a Trillion Cells per Mouse Click

TL;DR

This paper introduces PowerDrill's column-oriented datastore, which combines advanced data structures and partitioning techniques to enable interactive querying of datasets with trillions of cells per user click.

Contribution

The paper presents a novel columnar datastore design that significantly reduces memory usage and increases query speed for large-scale data analysis.

Findings

Achieves processing of a trillion cells per mouse click.

Provides large speed-ups over previous systems.

Enables highly interactive web-based data exploration.

Abstract

Column-oriented database systems have been a real game changer for the industry in recent years. Highly tuned and performant systems have evolved that provide users with the possibility of answering ad hoc queries over large datasets in an interactive manner. In this paper we present the column-oriented datastore developed as one of the central components of PowerDrill. It combines the advantages of columnar data layout with other known techniques (such as using composite range partitions) and extensive algorithmic engineering on key data structures. The main goal of the latter being to reduce the main memory footprint and to increase the efficiency in processing typical user queries. In this combination we achieve large speed-ups. These enable a highly interactive Web UI where it is common that a single mouse click leads to processing a trillion values in the underlying dataset.