TOPJoin: A Context-Aware Multi-Criteria Approach for Joinable Column Search

TL;DR

TOPJoin introduces a context-aware, multi-criteria method for finding joinable columns in enterprise data lakes, outperforming existing approaches by considering the query context for more accurate table joins.

Contribution

The paper proposes TOPJoin, a novel multi-criteria approach that incorporates query context to improve joinable column search in enterprise data lakes.

Findings

TOPJoin outperforms baseline methods on academic and real-world benchmarks.

Context-awareness significantly improves joinability detection accuracy.

The approach effectively combines multiple criteria for more precise joinable column identification.

Abstract

One of the major challenges in enterprise data analysis is the task of finding joinable tables that are conceptually related and provide meaningful insights. Traditionally, joinable tables have been discovered through a search for similar columns, where two columns are considered similar syntactically if there is a set overlap or they are considered similar semantically if either the column embeddings or value embeddings are closer in the embedding space. However, for enterprise data lakes, column similarity is not sufficient to identify joinable columns and tables. The context of the query column is important. Hence, in this work, we first define context-aware column joinability. Then we propose a multi-criteria approach, called TOPJoin, for joinable column search. We evaluate TOPJoin against existing join search baselines over one academic and one real-world join search benchmark. Through experiments, we find that TOPJoin performs better on both benchmarks than the baselines.