On the Structural Generalization in Text-to-SQL

TL;DR

This paper investigates the structural generalization in text-to-SQL models, highlighting dataset limitations and demonstrating that current models overfit to specific pattern templates, which hampers their ability to generalize to diverse database schemas.

Contribution

It introduces a framework for generating synthetic data to evaluate structural generalization and reveals the overfitting issue in current text-to-SQL models.

Findings

Significant performance drops on synthetic samples show limited structural generalization.

Current datasets are too templated, restricting the study of structural diversity.

Overfitting to (NL, SQL) patterns is a key challenge in the field.

Abstract

Exploring the generalization of a text-to-SQL parser is essential for a system to automatically adapt the real-world databases. Previous works provided investigations focusing on lexical diversity, including the influence of the synonym and perturbations in both natural language questions and databases. However, research on the structure variety of database schema~(DS) is deficient. Specifically, confronted with the same input question, the target SQL is probably represented in different ways when the DS comes to a different structure. In this work, we provide in-deep discussions about the structural generalization of text-to-SQL tasks. We observe that current datasets are too templated to study structural generalization. To collect eligible test data, we propose a framework to generate novel text-to-SQL data via automatic and synchronous (DS, SQL) pair altering. In the experiments, significant performance reduction when evaluating well-trained text-to-SQL models on the synthetic samples demonstrates the limitation of current research regarding structural generalization. According to comprehensive analysis, we suggest the practical reason is the overfitting of (NL, SQL) patterns.