Calibrating LLMs with Semantic-level Reward

TL;DR

This paper introduces a semantic-level reward framework called CSR to improve the calibration of large language models, ensuring their confidence estimates are more reliable across various tasks and distributions.

Contribution

CSR directly calibrates LLMs in semantic space, overcoming verbalized confidence limitations and enhancing calibration metrics across multiple datasets and model families.

Findings

CSR reduces ECE by up to 40% compared to baselines.

CSR improves AUROC by up to 31% over verbalized-confidence methods.

Calibration behavior generalizes robustly across datasets and model types.

Abstract

As large language models (LLMs) are deployed in consequential settings such as medical question answering and legal reasoning, the ability to estimate when their outputs are likely to be correct is essential for safe and reliable use, requiring well-calibrated uncertainty. Standard reinforcement learning with verifiable rewards (RLVR) trains models with a binary correctness reward that is indifferent to confidence, providing no penalty for confident but wrong predictions and thereby degrading calibration. Recent work addresses this by training models to produce verbalized confidence scores alongside answers and rewarding agreement with correctness. However, verbalized confidence is calibrated at the token level and thus exhibits inconsistency across textual variations with same semantic meaning. We propose \textbf{Calibration with Semantic Reward (CSR)}, a framework that calibrates language models directly in semantic space without a verbalized confidence interface. CSR combines the correctness reward with a novel semantic calibration reward that encourages exploitation among correct rollouts by promoting semantic agreement, and exploration among incorrect ones by discouraging spurious consistency. Experiments across three model families on HotpotQA (in-distribution) and TriviaQA, MSMARCO, and NQ-Open (out-of-distribution) show that CSR consistently achieves lower ECE and higher AUROC than verbalized-confidence baselines across nearly all settings, reducing ECE by up to $40\%$ and improving AUROC by up to $31\%$ over verbalized-confidence baselines, with calibration behavior generalizing robustly across all four evaluation settings.