Confidence Estimation for Error Detection in Text-to-SQL Systems

TL;DR

This paper explores confidence estimation and selective classification techniques to improve error detection and calibration in Text-to-SQL systems, enhancing their robustness and interpretability.

Contribution

It introduces entropy-based selective classifiers and calibration methods to improve error detection and confidence alignment in Text-to-SQL models, with empirical evaluation across different architectures.

Findings

Encoder-decoder T5 is better calibrated than GPT-4 and Llama 3.

Selective classifiers effectively detect errors in irrelevant questions.

Calibration techniques improve model confidence and accuracy alignment.

Abstract

Text-to-SQL enables users to interact with databases through natural language, simplifying the retrieval and synthesis of information. Despite the success of large language models (LLMs) in converting natural language questions into SQL queries, their broader adoption is limited by two main challenges: achieving robust generalization across diverse queries and ensuring interpretative confidence in their predictions. To tackle these issues, our research investigates the integration of selective classifiers into Text-to-SQL systems. We analyse the trade-off between coverage and risk using entropy based confidence estimation with selective classifiers and assess its impact on the overall performance of Text-to-SQL models. Additionally, we explore the models' initial calibration and improve it with calibration techniques for better model alignment between confidence and accuracy. Our experimental results show that encoder-decoder T5 is better calibrated than in-context-learning GPT 4 and decoder-only Llama 3, thus the designated external entropy-based selective classifier has better performance. The study also reveal that, in terms of error detection, selective classifier with a higher probability detects errors associated with irrelevant questions rather than incorrect query generations.