NOCAP: Near-Optimal Correlation-Aware Partitioning Joins

TL;DR

NOCAP is a new partitioning join algorithm that optimally exploits correlation skew in join attributes, significantly improving performance over existing methods under various memory constraints.

Contribution

The paper introduces NOCAP, a correlation-aware partitioning algorithm that achieves near-optimal join performance by tailoring partitioning to attribute correlation distributions.

Findings

NOCAP outperforms state-of-the-art algorithms by up to 30% on skewed data.

NOCAP is up to 4 times faster than Grace Hash Join.

NOCAP maintains high performance across a wide range of memory budgets.

Abstract

Storage-based joins are still commonly used today because the memory budget does not always scale with the data size. One of the many join algorithms developed that has been widely deployed and proven to be efficient is the Hybrid Hash Join (HHJ), which is designed to exploit any available memory to maximize the data that is joined directly in memory. However, HHJ cannot fully exploit detailed knowledge of the join attribute correlation distribution. In this paper, we show that given a correlation skew in the join attributes, HHJ partitions data in a suboptimal way. To do that, we derive the optimal partitioning using a new cost-based analysis of partitioning-based joins that is tailored for primary key - foreign key (PK-FK) joins, one of the most common join types. This optimal partitioning strategy has a high memory cost, thus, we further derive an approximate algorithm that has tunable memory cost and leads to near-optimal results. Our algorithm, termed NOCAP (Near-Optimal Correlation-Aware Partitioning) join, outperforms the state-of-the-art for skewed correlations by up to $30\%$, and the textbook Grace Hash Join by up to $4\times$. Further, for a limited memory budget, NOCAP outperforms HHJ by up to $10\%$, even for uniform correlation. Overall, NOCAP dominates state-of-the-art algorithms and mimics the best algorithm for a memory budget varying from below $\sqrt{\|\text{relation}\|}$ to more than $\|\text{relation}\|$.