A Study of In-Context-Learning-Based Text-to-SQL Errors

TL;DR

This paper provides a comprehensive analysis of errors in in-context learning-based text-to-SQL models, identifying error types, limitations of existing repairs, and proposing a new effective error detection and repair framework called MapleRepair.

Contribution

It is the first extensive study of text-to-SQL errors in ICL-based models and introduces MapleRepair, a novel framework that improves repair accuracy and efficiency.

Findings

Errors are widespread and categorized into 7 types.

Existing repairs have limited effectiveness and high computational costs.

MapleRepair outperforms existing methods in repair accuracy and efficiency.

Abstract

Large language models (LLMs) have been adopted to perform text-to-SQL tasks, utilizing their in-context learning (ICL) capability to translate natural language questions into structured query language (SQL). However, such a technique faces correctness problems and requires efficient repairing solutions. In this paper, we conduct the first comprehensive study of text-to-SQL errors. Our study covers four representative ICL-based techniques, five basic repairing methods, two benchmarks, and two LLM settings. We find that text-to-SQL errors are widespread and summarize 29 error types of 7 categories. We also find that existing repairing attempts have limited correctness improvement at the cost of high computational overhead with many mis-repairs. Based on the findings, we propose MapleRepair, a novel text-to-SQL error detection and repairing framework. The evaluation demonstrates that MapleRepair outperforms existing solutions by repairing 13.8% more queries with neglectable mis-repairs and 67.4% less overhead.