Text-to-SQL Calibration: No Need to Ask -- Just Rescale Model Probabilities

TL;DR

This paper demonstrates that simply rescaling the model's full-sequence probability effectively calibrates confidence in generated SQL queries, outperforming more complex recent methods across multiple benchmarks and models.

Contribution

The study reveals that a basic probability rescaling approach surpasses recent self-checking calibration techniques in Text-to-SQL tasks.

Findings

Full-sequence probability rescaling outperforms recent calibration methods.

Simple baseline achieves better confidence calibration.

Evaluation across multiple benchmarks confirms effectiveness.

Abstract

Calibration is crucial as large language models (LLMs) are increasingly deployed to convert natural language queries into SQL for commercial databases. In this work, we investigate calibration techniques for assigning confidence to generated SQL queries. We show that a straightforward baseline -- deriving confidence from the model's full-sequence probability -- outperforms recent methods that rely on follow-up prompts for self-checking and confidence verbalization. Our comprehensive evaluation, conducted across two widely-used Text-to-SQL benchmarks and multiple LLM architectures, provides valuable insights into the effectiveness of various calibration strategies.