Nexus: Inferring Join Graphs from Metadata Alone via Iterative Low-Rank Matrix Completion

TL;DR

Nexus is a novel method that infers join graphs from metadata alone by leveraging low-rank matrix completion and LLMs, significantly improving accuracy and speed in large schemas.

Contribution

The paper introduces Nexus, a new approach that formulates join graph inference as a low-rank matrix completion problem and enhances it with an EM algorithm using LLMs, addressing data access constraints.

Findings

Nexus outperforms existing methods on multiple datasets.

It effectively infers join graphs using only metadata.

The approach achieves up to 6x speedup in fast mode.

Abstract

Automatically inferring join relationships is a critical task for effective data discovery, integration, querying and reuse. However, accurately and efficiently identifying these relationships in large and complex schemas can be challenging, especially in enterprise settings where access to data values is constrained. In this paper, we introduce the problem of join graph inference when only metadata is available. We conduct an empirical study on a large number of real-world schemas and observe that join graphs when represented as adjacency matrices exhibit two key properties: high sparsity and low-rank structure. Based on these novel observations, we formulate join graph inference as a low-rank matrix completion problem and propose Nexus, an end-to-end solution using only metadata. To further enhance accuracy, we propose a novel Expectation-Maximization algorithm that alternates between low-rank matrix completion and refining join candidate probabilities by leveraging Large Language Models. Our extensive experiments demonstrate that Nexus outperforms existing methods by a significant margin on four datasets including a real-world production dataset. Additionally, Nexus can operate in a fast mode, providing comparable results with up to 6x speedup, offering a practical and efficient solution for real-world deployments.