Text-to-SQL Domain Adaptation via Human-LLM Collaborative Data Annotation

TL;DR

This paper introduces SQLsynth, a human-LLM collaborative system that efficiently generates high-quality, diverse text-to-SQL datasets, addressing domain adaptation challenges in real-world applications.

Contribution

SQLsynth is a novel human-in-the-loop annotation system that accelerates data creation and improves quality for domain-specific text-to-SQL models.

Findings

SQLsynth significantly speeds up data annotation.

It reduces cognitive load for annotators.

Datasets produced are more accurate, natural, and diverse.

Abstract

Text-to-SQL models, which parse natural language (NL) questions to executable SQL queries, are increasingly adopted in real-world applications. However, deploying such models in the real world often requires adapting them to the highly specialized database schemas used in specific applications. We find that existing text-to-SQL models experience significant performance drops when applied to new schemas, primarily due to the lack of domain-specific data for fine-tuning. This data scarcity also limits the ability to effectively evaluate model performance in new domains. Continuously obtaining high-quality text-to-SQL data for evolving schemas is prohibitively expensive in real-world scenarios. To bridge this gap, we propose SQLsynth, a human-in-the-loop text-to-SQL data annotation system. SQLsynth streamlines the creation of high-quality text-to-SQL datasets through human-LLM collaboration in a structured workflow. A within-subjects user study comparing SQLsynth with manual annotation and ChatGPT shows that SQLsynth significantly accelerates text-to-SQL data annotation, reduces cognitive load, and produces datasets that are more accurate, natural, and diverse. Our code is available at https://github.com/magic-YuanTian/SQLsynth.