ConfTuner: Training Large Language Models to Express Their Confidence Verbally

TL;DR

ConfTuner is a novel fine-tuning method for large language models that improves their ability to express calibrated confidence verbally, enhancing trustworthiness and performance in high-stakes applications without requiring ground-truth confidence labels.

Contribution

It introduces a proper scoring rule-based loss function, the tokenized Brier score, for calibrating LLMs' verbal confidence without ground-truth scores, applicable to diverse models and tasks.

Findings

Improves calibration across multiple reasoning tasks

Generalizes to black-box models like GPT-4o

Enhances downstream self-correction and cascade performance

Abstract

Large Language Models (LLMs) are increasingly deployed in high-stakes domains such as science, law, and healthcare, where accurate expressions of uncertainty are essential for reliability and trust. However, current LLMs are often observed to generate incorrect answers with high confidence, a phenomenon known as "overconfidence". Recent efforts have focused on calibrating LLMs' verbalized confidence: i.e., their expressions of confidence in text form, such as "I am 80% confident that...". Existing approaches either rely on prompt engineering or fine-tuning with heuristically generated uncertainty estimates, both of which have limited effectiveness and generalizability. Motivated by the notion of proper scoring rules for calibration in classical machine learning models, we introduce ConfTuner, a simple and efficient fine-tuning method that introduces minimal overhead and does not require ground-truth confidence scores or proxy confidence estimates. ConfTuner relies on a new loss function, tokenized Brier score, which we theoretically prove to be a proper scoring rule, intuitively meaning that it "correctly incentivizes the model to report its true probability of being correct". ConfTuner improves calibration across diverse reasoning tasks and generalizes to black-box models such as GPT-4o. Our results further show that better-calibrated confidence enables downstream gains in self-correction and model cascade, advancing the development of trustworthy LLM systems. The code is available at https://github.com/liushiliushi/ConfTuner.