Beyond SELECT: A Comprehensive Taxonomy-Guided Benchmark for Real-World Text-to-SQL Translation

TL;DR

This paper introduces a taxonomy-guided approach to create a diverse and comprehensive Text-to-SQL benchmark dataset, SQL-Synth, to better evaluate and improve LLM performance on real-world applications.

Contribution

It proposes a new taxonomy for Text-to-SQL tasks, uses it to synthesize a diverse dataset with LLMs, and demonstrates its effectiveness over existing datasets.

Findings

Existing datasets lack diversity and coverage.

SQL-Synth outperforms previous benchmarks in diversity.

Fine-tuning improves LLM performance on complex scenarios.

Abstract

Text-to-SQL datasets are essential for training and evaluating text-to-SQL models, but existing datasets often suffer from limited coverage and fail to capture the diversity of real-world applications. To address this, we propose a novel taxonomy for text-to-SQL classification based on dimensions including core intents, statement types, syntax structures, and key actions. Using this taxonomy, we evaluate widely used public text-to-SQL datasets (e.g., Spider and Bird) and reveal limitations in their coverage and diversity. We then introduce a taxonomy-guided dataset synthesis pipeline, yielding a new dataset named SQL-Synth. This approach combines the taxonomy with Large Language Models (LLMs) to ensure the dataset reflects the breadth and complexity of real-world text-to-SQL applications. Extensive analysis and experimental results validate the effectiveness of our taxonomy, as SQL-Synth exhibits greater diversity and coverage compared to existing benchmarks. Moreover, we uncover that existing LLMs typically fall short in adequately capturing the full range of scenarios, resulting in limited performance on SQL-Synth. However, fine-tuning can substantially improve their performance in these scenarios. The proposed taxonomy has significant potential impact, as it not only enables comprehensive analysis of datasets and the performance of different LLMs, but also guides the construction of training data for LLMs.