Metadata-driven Table Union Search: Leveraging Semantics for Restricted Access Data Integration

TL;DR

This paper presents a metadata-driven approach for table union search in restricted access data, achieving high accuracy and outperforming benchmarks, thus enabling secure data integration without compromising privacy.

Contribution

It introduces a novel method leveraging metadata alone for union search in restricted data, addressing privacy concerns absent in previous open-data reliant solutions.

Findings

Achieves 81% accuracy in unionability detection

Outperforms existing benchmarks in precision and recall

Supports privacy-preserving data discovery

Abstract

Over the past decade, the Table Union Search (TUS) task has aimed to identify unionable tables within data lakes to improve data integration and discovery. While numerous solutions and approaches have been introduced, they primarily rely on open data, making them not applicable to restricted access data, such as medical records or government statistics, due to privacy concerns. Restricted data can still be shared through metadata, which ensures confidentiality while supporting data reuse. This paper explores how TUS can be computed on restricted access data using metadata alone. We propose a method that achieves 81% accuracy in unionability and outperforms existing benchmarks in precision and recall. Our results highlight the potential of metadata-driven approaches for integrating restricted data, facilitating secure data discovery in privacy-sensitive domains. This aligns with the FAIR principles, by ensuring data is Findable, Accessible, Interoperable, and Reusable while preserving confidentiality.