Improving Text-to-SQL Evaluation Methodology

TL;DR

This paper critically evaluates current text-to-SQL evaluation practices, proposing standardized datasets, new evaluation splits, and emphasizing the importance of variable anonymization to better measure system generalization and real-world applicability.

Contribution

It introduces improved datasets, a new evaluation split, and highlights the significance of variable anonymization for more realistic assessment of text-to-SQL systems.

Findings

Standardized and improved datasets released for evaluation.

A new dataset split to better test generalization.

Highlighting the importance of variable anonymization in evaluation.

Abstract

To be informative, an evaluation must measure how well systems generalize to realistic unseen data. We identify limitations of and propose improvements to current evaluations of text-to-SQL systems. First, we compare human-generated and automatically generated questions, characterizing properties of queries necessary for real-world applications. To facilitate evaluation on multiple datasets, we release standardized and improved versions of seven existing datasets and one new text-to-SQL dataset. Second, we show that the current division of data into training and test sets measures robustness to variations in the way questions are asked, but only partially tests how well systems generalize to new queries; therefore, we propose a complementary dataset split for evaluation of future work. Finally, we demonstrate how the common practice of anonymizing variables during evaluation removes an important challenge of the task. Our observations highlight key difficulties, and our methodology enables effective measurement of future development.