Balancing Classification and Calibration Performance in Decision-Making LLMs via Calibration Aware Reinforcement Learning

TL;DR

This paper investigates calibration issues in large language models during decision-making tasks, comparing supervised fine-tuning and reinforcement learning, and proposes a calibration-aware reinforcement learning method to improve confidence estimates without sacrificing accuracy.

Contribution

It introduces a calibration-aware reinforcement learning approach that enhances model calibration while maintaining high task performance.

Findings

RLVR produces overconfident models despite high performance.

SFT yields better calibration, even under distribution shift.

The proposed method reduces ECE scores by up to 9 points.

Abstract

Large language models (LLMs) are increasingly deployed in decision-making tasks, where not only accuracy but also reliable confidence estimates are essential. Well-calibrated confidence enables downstream systems to decide when to trust a model and when to defer to fallback mechanisms. In this work, we conduct a systematic study of calibration in two widely used fine-tuning paradigms: supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). We show that while RLVR improves task performance, it produces extremely overconfident models, whereas SFT yields substantially better calibration, even under distribution shift, though with smaller performance gains. Through targeted experiments, we diagnose RLVR's failure, showing that decision tokens act as extraction steps of the decision in reasoning traces and do not carry confidence information, which prevents reinforcement learning from surfacing calibrated alternatives. Based on this insight, we propose a calibration-aware reinforcement learning formulation that directly adjusts decision-token probabilities. Our method preserves RLVR's accuracy level while mitigating overconfidence, reducing ECE scores up to 9 points.