Synthetic SQL Column Descriptions and Their Impact on Text-to-SQL Performance

TL;DR

This paper investigates using large language models to automatically generate detailed SQL column descriptions to improve text-to-SQL performance, revealing that richer metadata enhances model accuracy especially with larger models.

Contribution

It introduces a dataset of SQL column descriptions, evaluates LLMs for description generation, and demonstrates that generated metadata improves text-to-SQL accuracy, surpassing manual descriptions in some cases.

Findings

LLMs struggle with ambiguous columns, needing expert input.

Generated descriptions improve text-to-SQL performance, especially for large models.

Qwen2 descriptions outperform manual gold descriptions, benefiting from detailed metadata.

Abstract

Relational databases often suffer from uninformative descriptors of table contents, such as ambiguous columns and hard-to-interpret values, impacting both human users and text-to-SQL models. In this paper, we explore the use of large language models (LLMs) to automatically generate detailed natural language descriptions for SQL database columns, aiming to improve text-to-SQL performance and automate metadata creation. We create a dataset of gold column descriptions based on the BIRD-Bench benchmark, manually refining its column descriptions and creating a taxonomy for categorizing column difficulty. We then evaluate several different LLMs in generating column descriptions across the columns and different difficulties in the dataset, finding that models unsurprisingly struggle with columns that exhibit inherent ambiguity, highlighting the need for manual expert input. We also find that incorporating such generated column descriptions consistently enhances text-to-SQL model performance, particularly for larger models like GPT-4o, Qwen2 72B and Mixtral 22Bx8. Notably, Qwen2-generated descriptions, containing by annotators deemed superfluous information, outperform manually curated gold descriptions, suggesting that models benefit from more detailed metadata than humans expect. Future work will investigate the specific features of these high-performing descriptions and explore other types of metadata, such as numerical reasoning and synonyms, to further improve text-to-SQL systems. The dataset, annotations and code will all be made available.