MT-Teql: Evaluating and Augmenting Consistency of Text-to-SQL Models with Metamorphic Testing
Pingchuan Ma, Shuai Wang

TL;DR
MT-Teql is a metamorphic testing framework that evaluates and improves the consistency of text-to-SQL models against linguistic and schema variations, significantly reducing errors and enhancing robustness.
Contribution
The paper introduces a model-agnostic metamorphic testing framework for systematically evaluating and augmenting text-to-SQL models' consistency, exposing errors, and boosting robustness.
Findings
Exposed thousands of prediction errors in SOTA models.
Enriched datasets by an order of magnitude.
Reduced over 40% inconsistency errors without accuracy loss.
Abstract
Text-to-SQL is a task to generate SQL queries from human utterances. However, due to the variation of natural language, two semantically equivalent utterances may appear differently in the lexical level. Likewise, user preferences (e.g., the choice of normal forms) can lead to dramatic changes in table structures when expressing conceptually identical schemas. Envisioning the general difficulty for text-to-SQL models to preserve prediction consistency against linguistic and schema variations, we propose MT-Teql, a Metamorphic Testing-based framework for systematically evaluating and augmenting the consistency of TExt-to-SQL models. Inspired by the principles of software metamorphic testing, MT-Teql delivers a model-agnostic framework which implements a comprehensive set of metamorphic relations (MRs) to conduct semantics-preserving transformations toward utterances and schemas. Model…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsScientific Computing and Data Management · Topic Modeling · Advanced Database Systems and Queries
