Creating and Querying Data Cubes in Python using pyCube

TL;DR

This paper introduces pyCube, a Python-based tool for creating and querying data cubes from relational databases, which outperforms pandas implementations in speed and memory usage, facilitating data analysis for data scientists.

Contribution

The paper presents pyCube, a novel Python library that simplifies data cube creation and querying, optimized for data scientists and evaluated on standard benchmarks.

Findings

pyCube significantly outperforms pandas in runtime

pyCube uses less memory than pandas implementations

pyCube is easier to read and write for users

Abstract

Data cubes are used for analyzing large data sets usually contained in data warehouses. The most popular data cube tools use graphical user interfaces (GUI) to do the data analysis. Traditionally this was fine since data analysts were not expected to be technical people. However, in the subsequent decades the data landscape changed dramatically requiring companies to employ large teams of highly technical data scientists in order to manage and use the ever increasing amount of data. These data scientists generally use tools like Python, interactive notebooks, pandas, etc. while modern data cube tools are still GUI based. This paper proposes a Python-based data cube tool called pyCube. pyCube is able to semi-automatically create data cubes for data stored in an RDBMS and manages the data cube metadata. pyCube's programmatic interface enables data scientist to query data cubes by specifying the expected metadata of the result. pyCube is experimentally evaluated on Star Schema Benchmark (SSB). The results show that pyCube vastly outperforms different implementations of SSB queries in pandas in both runtime and memory while being easier to read and write.