On the Structural Generalization in Text-to-SQL
Jieyu Li, Lu Chen, Ruisheng Cao, Su Zhu, Hongshen Xu, Zhi Chen,, Hanchong Zhang, Kai Yu

TL;DR
This paper investigates the structural generalization in text-to-SQL models, highlighting dataset limitations and demonstrating that current models overfit to specific pattern templates, which hampers their ability to generalize to diverse database schemas.
Contribution
It introduces a framework for generating synthetic data to evaluate structural generalization and reveals the overfitting issue in current text-to-SQL models.
Findings
Significant performance drops on synthetic samples show limited structural generalization.
Current datasets are too templated, restricting the study of structural diversity.
Overfitting to (NL, SQL) patterns is a key challenge in the field.
Abstract
Exploring the generalization of a text-to-SQL parser is essential for a system to automatically adapt the real-world databases. Previous works provided investigations focusing on lexical diversity, including the influence of the synonym and perturbations in both natural language questions and databases. However, research on the structure variety of database schema~(DS) is deficient. Specifically, confronted with the same input question, the target SQL is probably represented in different ways when the DS comes to a different structure. In this work, we provide in-deep discussions about the structural generalization of text-to-SQL tasks. We observe that current datasets are too templated to study structural generalization. To collect eligible test data, we propose a framework to generate novel text-to-SQL data via automatic and synchronous (DS, SQL) pair altering. In the experiments,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Topic Modeling · Semantic Web and Ontologies
MethodsTest
