AnnotatedTables: A Large Tabular Dataset with Language Model Annotations

TL;DR

This paper introduces AnnotatedTables, a large dataset of over 32,000 databases with LLM-generated annotations, demonstrating the potential of language models to automate tabular data annotation and support diverse research tasks.

Contribution

The paper presents a scalable methodology for annotating large tabular datasets with language models, including SQL and input-target column annotations, and releases the AnnotatedTables dataset.

Findings

LLMs can generate accurate SQL annotations for large datasets.

LLMs can translate SQL to Rel programs with few-shot prompting.

TabPFN classifier performs comparably to AutoML on annotated tables.

Abstract

Tabular data is ubiquitous in real-world applications and abundant on the web, yet its annotation has traditionally required human labor, posing a significant scalability bottleneck for tabular machine learning. Our methodology can successfully annotate a large amount of tabular data and can be flexibly steered to generate various types of annotations based on specific research objectives, as we demonstrate with SQL annotation and input-target column annotation as examples. As a result, we release AnnotatedTables, a collection of 32,119 databases with LLM-generated annotations. The dataset includes 405,616 valid SQL programs, making it the largest SQL dataset with associated tabular data that supports query execution. To further demonstrate the value of our methodology and dataset, we perform two follow-up research studies. 1) We investigate whether LLMs can translate SQL programs to Rel programs, a database language previously unknown to LLMs, while obtaining the same execution results. Using our Incremental Prompt Engineering methods based on execution feedback, we show that LLMs can produce adequate translations with few-shot learning. 2) We evaluate the performance of TabPFN, a recent neural tabular classifier trained on Bayesian priors, on 2,720 tables with input-target columns identified and annotated by LLMs. On average, TabPFN performs on par with the baseline AutoML method, though the relative performance can vary significantly from one data table to another, making both models viable for practical applications depending on the situation. Our findings underscore the potential of LLMs in automating the annotation of large volumes of diverse tabular data.