LAKEGEN: A LLM-based Tabular Corpus Generator for Evaluating Dataset Discovery in Data Lakes

TL;DR

This paper introduces LAKEGEN, a novel LLM-based tool for generating realistic, domain-specific tabular datasets with joinability annotations, to improve the evaluation of dataset discovery methods in data lakes.

Contribution

LAKEGEN leverages large language models to create diverse, high-quality tabular benchmarks with annotated joinability, addressing limitations of existing datasets for dataset discovery evaluation.

Findings

LAKEGEN produces realistic, domain-specific tables with annotated joinability.

Generated datasets improve the evaluation of dataset discovery methods.

The approach addresses limitations of current open data corpora.

Abstract

How to generate a large, realistic set of tables along with joinability relationships, to stress-test dataset discovery methods? Dataset discovery methods aim to automatically identify related data assets in a data lake. The development and evaluation of such solutions for customers from a wide range of business domains, relies on diverse, high quality and domain-specific tabular benchmarks. Large language models (LLMs) are trained on a wide variety of text data, which can provide a strong foundation of general and domain-specific knowledge. In this paper, we ask the question -- \textit{can we leverage LLMs to generate a tabular benchmark adequate for evaluating the dataset discovery solutions?} In particular, we focus on the task of finding joinable tables which is the cornerstone of virtually every dataset discovery method. Current corpora for evaluating dataset discovery methods are mainly based on subsets of open data, and they suffer from three important issues: $i)$ they focus on very common and generic data types (e.g., address, id, name, etc.); $ii)$ they do not contain human-annotated column pairs; instead, practitioners synthesize ground truth using table splits (e.g., horizontal for table union search and vertical ones for joinability) and $iii)$ they do not focus on semantic column relationships.