Less Approximates More: Harmonizing Performance and Confidence Faithfulness via Hybrid Post-Training for High-Stakes Tasks

TL;DR

This paper introduces HyTuning, a hybrid post-training method that enhances confidence faithfulness and accuracy in large language models for high-stakes tasks by adaptively balancing reasoning distillation and reinforcement learning.

Contribution

It proposes a novel hybrid post-training framework using a Progressive Reasoning Gain metric to improve confidence faithfulness with limited supervision.

Findings

HyTuning improves accuracy on multiple benchmarks.

It achieves better confidence faithfulness under limited supervision.

The approach supports a 'Less Approximates More' effect.

Abstract

Large language models are increasingly deployed in high-stakes tasks, where confident yet incorrect inferences may cause severe real-world harm, bringing the previously overlooked issue of confidence faithfulness back to the forefront. A promising solution is to jointly optimize unsupervised Reinforcement Learning from Internal Feedback (RLIF) with reasoning-trace-guided Reasoning Distillation (RD), which may face three persistent challenges: scarcity of high-quality training corpora, factually unwarranted overconfidence and indiscriminate fusion that amplifies erroneous updates. Inspired by the human confidence accumulation from uncertainty to certainty, we propose Progressive Reasoning Gain (PRG) to measure whether reasoning steps progressively strengthen support for the final answer. Furthermore, we introduce HyTuning, a hybrid post-training framework that adaptively reweights RD and RLIF via a PRG-style metric, using scarce supervised reasoning traces as a stable anchor while exploiting abundant unlabeled queries for scalability. Experiments on several domain-specific and general benchmarks demonstrate that HyTuning improves accuracy while achieving confidence faithfulness under limited supervision, supporting a practical "Less Approximates More" effect.