FREYJA: Efficient Join Discovery in Data Lakes

TL;DR

FREYJA is a data discovery system for data lakes that efficiently identifies relevant join candidates using a novel, scalable join quality metric based on data profiles, achieving high accuracy with significantly reduced computation.

Contribution

FREYJA introduces a scalable join quality measure and a predictive model leveraging data profiles, outperforming existing methods in efficiency while maintaining accuracy.

Findings

FREYJA matches state-of-the-art accuracy in join discovery.

It reduces execution times by several orders of magnitude.

The system effectively explores large data lakes for downstream tasks.

Abstract

Data lakes are massive repositories of raw and heterogeneous data, designed to meet the requirements of modern data storage. Nonetheless, this same philosophy increases the complexity of performing discovery tasks to find relevant data for subsequent processing. As a response to these growing challenges, we present FREYJA, a modern data discovery system capable of effectively exploring data lakes, aimed at finding candidates to perform joins and increase the number of attributes for downstream tasks. More precisely, we want to compute rankings that sort potential joins by their relevance. Modern mechanisms apply advanced table representation learning (TRL) techniques to yield accurate joins. Yet, this incurs high computational costs when dealing with elevated volumes of data. In contrast to the state-of-the-art, we adopt a novel notion of join quality tailored to data lakes, which leverages syntactic measurements while achieving accuracy comparable to that of TRL approaches. To obtain this metric in a scalable manner we train a general purpose predictive model. Predictions are based, rather than on large-scale datasets, on data profiles, succinct representations that capture the underlying characteristics of the data. Our experiments show that our system, FREYJA, matches the results of the state-of-the-art whilst reducing the execution times by several orders of magnitude.