QJoin: Transformation-aware Joinable Data Discovery Using Reinforcement Learning

TL;DR

QJoin is a reinforcement learning framework that improves joinable data discovery by learning transformation strategies and reusing them, leading to higher accuracy and efficiency in heterogeneous data repositories.

Contribution

It introduces a reinforcement learning approach with transfer and reuse mechanisms for transformation-aware join discovery in large, heterogeneous datasets.

Findings

Achieves 91.0% F1-score on AutoJoin Web benchmark.

Reduces join task runtime by up to 7.4%.

Effectively reuses learned transformations for similar tasks.

Abstract

Discovering which tables in large, heterogeneous repositories can be joined and by what transformations is a central challenge in data integration and data discovery. Traditional join discovery methods are largely designed for equi-joins, which assume that join keys match exactly or nearly so. These techniques, while efficient in clean, well-normalized databases, fail in open or federated settings where identifiers are inconsistently formatted, embedded, or split across multiple columns. Approximate or fuzzy joins alleviate minor string variations but cannot capture systematic transformations. We introduce QJoin, a reinforcement-learning framework that learns and reuses transformation strategies across join tasks. QJoin trains an agent under a uniqueness-aware reward that balances similarity with key distinctiveness, enabling it to explore concise, high-value transformation chains. To accelerate new joins, we introduce two reuse mechanisms: (i) agent transfer, which initializes new policies from pretrained agents, and (ii) transformation reuse, which caches successful operator sequences for similar column clusters. On the AutoJoin Web benchmark (31 table pairs), QJoin achieves an average F1-score of 91.0%. For 19,990 join tasks in NYC+Chicago open datasets, Qjoin reduces runtime by up to 7.4% (13,747 s) by using reusing. These results demonstrate that transformation learning and reuse can make join discovery both more accurate and more efficient.