Calibrating LLMs for Text-to-SQL Parsing by Leveraging Sub-clause Frequencies

TL;DR

This paper introduces a new calibration method for large language models in text-to-SQL tasks, using sub-clause frequency scores and multivariate Platt scaling to improve confidence estimates and error detection.

Contribution

It is the first to benchmark post-hoc calibration for LLM-based text-to-SQL and proposes leveraging SQL structure via SCF scores combined with MPS for better calibration.

Findings

SCF scores improve calibration accuracy.

MPS combined with SCF outperforms traditional Platt scaling.

Enhanced error detection in text-to-SQL tasks.

Abstract

While large language models (LLMs) achieve strong performance on text-to-SQL parsing, they sometimes exhibit unexpected failures in which they are confidently incorrect. Building trustworthy text-to-SQL systems thus requires eliciting reliable uncertainty measures from the LLM. In this paper, we study the problem of providing a calibrated confidence score that conveys the likelihood of an output query being correct. Our work is the first to establish a benchmark for post-hoc calibration of LLM-based text-to-SQL parsing. In particular, we show that Platt scaling, a canonical method for calibration, provides substantial improvements over directly using raw model output probabilities as confidence scores. Furthermore, we propose a method for text-to-SQL calibration that leverages the structured nature of SQL queries to provide more granular signals of correctness, named "sub-clause frequency" (SCF) scores. Using multivariate Platt scaling (MPS), our extension of the canonical Platt scaling technique, we combine individual SCF scores into an overall accurate and calibrated score. Empirical evaluation on two popular text-to-SQL datasets shows that our approach of combining MPS and SCF yields further improvements in calibration and the related task of error detection over traditional Platt scaling.