EasyTUS: A Comprehensive Framework for Fast and Accurate Table Union Search across Data Lakes

TL;DR

EasyTUS is a new framework that uses Large Language Models to efficiently and accurately find tables in data lakes that can be unioned with a given table, improving speed and precision.

Contribution

This work introduces EasyTUS, a novel LLM-based framework for scalable Table Union Search and a standardized benchmarking environment, TUSBench, for systematic evaluation.

Findings

Up to 34.3% improvement in MAP over state-of-the-art methods.

Up to 79.2x faster data preparation.

Up to 7.7x faster query processing.

Abstract

Data lakes enable easy maintenance of heterogeneous data in its native form. While this flexibility can accelerate data ingestion, it shifts the complexity of data preparation and query processing to data discovery tasks. One such task is Table Union Search (TUS), which identifies tables that can be unioned with a given input table. In this work, we present EasyTUS, a comprehensive framework that leverages Large Language Models (LLMs) to perform efficient and scalable Table Union Search across data lakes. EasyTUS implements the search pipeline as three modular steps: Table Serialization for consistent formatting and sampling, Table Representation that utilizes LLMs to generate embeddings, and Vector Search that leverages approximate nearest neighbor indexing for semantic matching. To enable reproducible and systematic evaluation, in this paper, we also introduce TUSBench, a novel standardized benchmarking environment within the EasyTUS framework. TUSBench supports unified comparisons across approaches and data lakes, promoting transparency and progress in the field. Our experiments using TUSBench show that EasyTUS consistently outperforms most of the state-of the-art approaches, achieving improvements in average of up to 34.3% in Mean Average Precision (MAP), up to 79.2x speedup in data preparation, and up to 7.7x faster query processing performance. Furthermore, EasyTUS maintains strong performance even in metadata-absent settings, highlighting its robustness and adaptability across data lakes.