Scalable Data Discovery Using Profiles

TL;DR

This paper introduces NextiaJD, a scalable, profile-based system for discovering joinable datasets at scale, achieving high accuracy and efficiency while reducing false positives compared to existing hash-based methods.

Contribution

It proposes a novel profile-based approach with a new join quality metric, enabling scalable and accurate dataset joinability prediction at large volumes.

Findings

NextiaJD matches hash-based methods in predictive performance.

It scales to larger data volumes efficiently.

It significantly reduces false positives at scale.

Abstract

We study the problem of discovering joinable datasets at scale. This is, how to automatically discover pairs of attributes in a massive collection of independent, heterogeneous datasets that can be joined. Exact (e.g., based on distinct values) and hash-based (e.g., based on locality-sensitive hashing) techniques require indexing the entire dataset, which is unattainable at scale. To overcome this issue, we approach the problem from a learning perspective relying on profiles. These are succinct representations that capture the underlying characteristics of the schemata and data values of datasets, which can be efficiently extracted in a distributed and parallel fashion. Profiles are then compared, to predict the quality of a join operation among a pair of attributes from different datasets. In contrast to the state-of-the-art, we define a novel notion of join quality that relies on a metric considering both the containment and cardinality proportions between candidate attributes. We implement our approach in a system called NextiaJD, and present extensive experiments to show the predictive performance and computational efficiency of our method. Our experiments show that NextiaJD obtains similar predictive performance to that of hash-based methods, yet we are able to scale-up to larger volumes of data. Also, NextiaJD generates a considerably less amount of false positives, which is a desirable feature at scale.