Scalable Relational Query Processing on Big Matrix Data

TL;DR

This paper introduces scalable relational query processing methods for large matrix data in distributed environments, significantly improving performance over existing systems by optimizing query plans and partitioning strategies.

Contribution

It develops novel algebraic transformations, a query optimizer, and partitioning schemes for efficient relational operations directly on big matrix data in distributed clusters.

Findings

Achieves up to 100x performance improvement over state-of-the-art systems.

Demonstrates effectiveness on real and synthetic datasets.

Prototypes in Apache Spark validate the approach.

Abstract

The use of large-scale machine learning methods is becoming ubiquitous in many applications ranging from business intelligence to self-driving cars. These methods require a complex computation pipeline consisting of various types of operations, e.g., relational operations for pre-processing or post-processing the dataset, and matrix operations for core model computations. Many existing systems focus on efficiently processing matrix-only operations, and assume that the inputs to the relational operators are already pre-computed and are materialized as intermediate matrices. However, the input to a relational operator may be complex in machine learning pipelines, and may involve various combinations of matrix operators. Hence, it is critical to realize scalable and efficient relational query processors that directly operate on big matrix data. This paper presents new efficient and scalable relational query processing techniques on big matrix data for in-memory distributed clusters. The proposed techniques leverage algebraic transformation rules to rewrite query execution plans into ones with lower computation costs. A distributed query plan optimizer exploits the sparsity-inducing property of merge functions as well as Bloom join strategies for efficiently evaluating various flavors of the join operation. Furthermore, optimized partitioning schemes for the input matrices are developed to facilitate the performance of join operations based on a cost model that minimizes the communication overhead.The proposed relational query processing techniques are prototyped in Apache Spark. Experiments on both real and synthetic data demonstrate that the proposed techniques achieve up to two orders of magnitude performance improvement over state-of-the-art systems on a wide range of applications.