CARE-RFT: Confidence-Anchored Reinforcement Finetuning for Reliable Reasoning in Large Language Models

TL;DR

CARE-RFT introduces a confidence-anchored regularization technique for reinforcement finetuning large language models, balancing reasoning ability with trustworthiness and calibration by adaptively penalizing exploration based on confidence.

Contribution

It proposes a novel skew reverse KL divergence regularization method that improves the trade-off between reasoning performance and trustworthiness in RFT.

Findings

CARE-RFT matches reasoning performance of unconstrained RFT.

It recovers trustworthiness and calibration of the base model.

The method is effective across multiple model scales.

Abstract

Reinforcement finetuning (RFT) has emerged as a powerful paradigm for unlocking reasoning capabilities in large language models. However, we identify a critical trade-off: while unconstrained RFT achieves strong reasoning performance, it severely compromises model trustworthiness by amplifying hallucination and worsening calibration; conversely, RKL-constrained RFT preserves trustworthiness but limits reasoning gains due to its unbounded penalty on exploratory deviations. To resolve this tension, we introduce CARE-RFT (Confidence-Anchored Regularized Reinforcement Finetuning), a novel method that replaces standard reverse KL regularization with a skew reverse KL divergence. CARE-RFT provides a confidence-sensitive penalty: it is bounded for confident, consistently rewarded explorations to enable reasoning, while unbounded elsewhere to preserve calibration. Extensive experiments across multiple model scales and RFT algorithms show that CARE-RFT achieves a superior balance, matching the reasoning performance of unconstrained RFT while recovering the trustworthiness and calibration of the base model. Our work establishes that careful, confidence-aware regularization is key to building both capable and trustworthy reasoning models.