Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback

TL;DR

This paper evaluates methods for obtaining well-calibrated confidence scores from RLHF-finetuned language models, finding that verbalized confidences are often more accurate than raw probabilities across several benchmarks.

Contribution

It provides a comprehensive assessment of confidence elicitation strategies for RLHF-LMs, highlighting the effectiveness of verbalized confidences over raw probabilities.

Findings

Verbalized confidences are better calibrated than conditional probabilities.

Using verbalized confidences reduces expected calibration error by about 50%.

Results are demonstrated on TriviaQA, SciQ, and TruthfulQA benchmarks.

Abstract

A trustworthy real-world prediction system should produce well-calibrated confidence scores; that is, its confidence in an answer should be indicative of the likelihood that the answer is correct, enabling deferral to an expert in cases of low-confidence predictions. Recent studies have shown that unsupervised pre-training produces large language models (LMs) whose conditional probabilities are remarkably well-calibrated. However, the most widely-used LMs are fine-tuned with reinforcement learning from human feedback (RLHF-LMs), and some studies have suggested that RLHF-LMs produce conditional probabilities that are very poorly calibrated. In light of this perceived weakness, we conduct a broad evaluation of methods for extracting confidence scores from RLHF-LMs. For RLHF-LMs such as ChatGPT, GPT-4, and Claude, we find that verbalized confidences emitted as output tokens are typically better-calibrated than the model's conditional probabilities on the TriviaQA, SciQ, and TruthfulQA benchmarks, often reducing the expected calibration error by a relative 50%.