LevelHeaded: Making Worst-Case Optimal Joins Work in the Common Case

TL;DR

LevelHeaded introduces a unified in-memory query engine using worst-case optimal joins, achieving high performance on both BI and LA workloads and bridging the gap between relational databases and specialized engines.

Contribution

The paper presents LevelHeaded, a novel query processing engine that applies worst-case optimal joins to unify BI and LA workloads with significant performance improvements.

Findings

Outperforms existing relational engines by orders of magnitude on LA benchmarks.

Achieves within 31% of specialized BI and LA solutions on their benchmarks.

Demonstrates the viability of a single architecture for diverse query types.

Abstract

Pipelines combining SQL-style business intelligence (BI) queries and linear algebra (LA) are becoming increasingly common in industry. As a result, there is a growing need to unify these workloads in a single framework. Unfortunately, existing solutions either sacrifice the inherent benefits of exclusively using a relational database (e.g. logical and physical independence) or incur orders of magnitude performance gaps compared to specialized engines (or both). In this work we study applying a new type of query processing architecture to standard BI and LA benchmarks. To do this we present a new in-memory query processing engine called LevelHeaded. LevelHeaded uses worst-case optimal joins as its core execution mechanism for both BI and LA queries. With LevelHeaded, we show how crucial optimizations for BI and LA queries can be captured in a worst-case optimal query architecture. Using these optimizations, LevelHeaded outperforms other relational database engines (LogicBlox, MonetDB, and HyPer) by orders of magnitude on standard LA benchmarks, while performing on average within 31% of the best-of-breed BI (HyPer) and LA (Intel MKL) solutions on their own benchmarks. Our results show that such a single query processing architecture is capable of delivering competitive performance on both BI and LA queries.