Reliable Text-to-SQL with Adaptive Abstention

TL;DR

This paper introduces RTS, a framework that improves the reliability of text-to-SQL systems by detecting errors, abstaining, and involving humans, especially focusing on schema linking with probabilistic guarantees.

Contribution

RTS is the first to incorporate adaptive abstention and human-in-the-loop mechanisms with probabilistic schema linking guarantees in text-to-SQL models.

Findings

Achieves near-perfect schema linking accuracy on BIRD benchmark.

Significantly improves robustness and reliability of text-to-SQL conversion.

Small models with RTS nearly match state-of-the-art larger models.

Abstract

Large language models (LLMs) have revolutionized natural language interfaces for databases, particularly in text-to-SQL conversion. However, current approaches often generate unreliable outputs when faced with ambiguity or insufficient context. We present Reliable Text-to-SQL (RTS), a novel framework that enhances query generation reliability by incorporating abstention and human-in-the-loop mechanisms. RTS focuses on the critical schema linking phase, which aims to identify the key database elements needed for generating SQL queries. It autonomously detects potential errors during the answer generation process and responds by either abstaining or engaging in user interaction. A vital component of RTS is the Branching Point Prediction (BPP) which utilizes statistical conformal techniques on the hidden layers of the LLM model for schema linking, providing probabilistic guarantees on schema linking accuracy. We validate our approach through comprehensive experiments on the BIRD benchmark, demonstrating significant improvements in robustness and reliability. Our findings highlight the potential of combining transparent-box LLMs with human-in-the-loop processes to create more robust natural language interfaces for databases. For the BIRD benchmark, our approach achieves near-perfect schema linking accuracy, autonomously involving a human when needed. Combined with query generation, we demonstrate that near-perfect schema linking and a small query generation model can almost match SOTA accuracy achieved with a model orders of magnitude larger than the one we use.