ORCE: Order-Aware Alignment of Verbalized Confidence in Large Language Models

TL;DR

This paper introduces a decoupled, order-aware framework for verbalized confidence calibration in large language models, improving confidence alignment without sacrificing answer accuracy.

Contribution

It proposes a novel method that separates answer generation from confidence estimation and uses rank-based reinforcement learning for better calibration.

Findings

Improves confidence calibration and failure prediction performance.

Largely preserves answer accuracy while enhancing confidence reliability.

Demonstrates effectiveness on reasoning and knowledge-intensive benchmarks.

Abstract

Large language models (LLMs) often produce answers with high certainty even when they are incorrect, making reliable confidence estimation essential for deployment in real-world scenarios. Verbalized confidence, where models explicitly state their confidence in natural language, provides a flexible and user-facing uncertainty signal that can be applied even when token logits are unavailable. However, existing verbalized-confidence methods often optimize answer generation and confidence generation jointly, which can cause confidence-alignment objectives to interfere with answer accuracy. In this work, we propose a decoupled and order-aware framework for verbalized confidence calibration. Our method first generates an answer and then estimates confidence conditioned on the fixed question--answer pair, allowing confidence optimization without directly perturbing the answer-generation process. To align confidence with correctness likelihood, we construct a sampling-based surrogate from multiple model completions and optimize rank-based reinforcement learning objectives that encourage responses with higher estimated correctness likelihood to receive higher verbalized confidence. Experiments on reasoning and knowledge-intensive benchmarks show that our method improves calibration and failure prediction performance while largely preserving answer accuracy. These results demonstrate that verbalized confidence can be more reliably aligned by decoupling confidence estimation from answer generation and optimizing the relative ordering of confidence across responses.