Confidence Estimation for Text-to-SQL in Large Language Models

TL;DR

This paper investigates confidence estimation methods for text-to-SQL tasks using large language models, comparing black-box and white-box strategies, and demonstrating the benefits of execution grounding and syntax-aware approaches.

Contribution

It introduces and evaluates confidence estimation techniques for LLM-generated SQL queries, highlighting the effectiveness of consistency-based and syntax-aware methods across domains.

Findings

Consistency-based black-box methods outperform others.

SQL-syntax-aware white-box approaches improve interpretability.

Execution grounding enhances confidence estimation accuracy.

Abstract

Confidence estimation for text-to-SQL aims to assess the reliability of model-generated SQL queries without having access to gold answers. We study this problem in the context of large language models (LLMs), where access to model weights and gradients is often constrained. We explore both black-box and white-box confidence estimation strategies, evaluating their effectiveness on cross-domain text-to-SQL benchmarks. Our evaluation highlights the superior performance of consistency-based methods among black-box models and the advantage of SQL-syntax-aware approaches for interpreting LLM logits in white-box settings. Furthermore, we show that execution-based grounding of queries provides a valuable supplementary signal, improving the effectiveness of both approaches.