Beyond Correctness: Confidence-Aware Reward Modeling for Enhancing Large Language Model Reasoning

TL;DR

This paper introduces a confidence-aware reward model that improves reasoning quality in large language models by penalizing low-confidence correct answers, leading to more consistent and logical STEM reasoning.

Contribution

The paper presents a novel confidence-based reward model that enhances reasoning in LLMs by addressing limitations of traditional rule-based reward methods.

Findings

Outperforms state-of-the-art reward models on STEM benchmarks

Promotes more robust and logically consistent reasoning

Effective in both static and RL training evaluations

Abstract

Recent advancements in large language models (LLMs) have shifted the post-training paradigm from traditional instruction tuning and human preference alignment toward reinforcement learning (RL) focused on reasoning capabilities. However, numerous technical reports indicate that purely rule-based reward RL frequently results in poor-quality reasoning chains or inconsistencies between reasoning processes and final answers, particularly when the base model is of smaller scale. During the RL exploration process, models might employ low-quality reasoning chains due to the lack of knowledge, occasionally producing correct answers randomly and receiving rewards based on established rule-based judges. This constrains the potential for resource-limited organizations to conduct direct reinforcement learning training on smaller-scale models. We propose a novel confidence-based reward model tailored for enhancing STEM reasoning capabilities. Unlike conventional approaches, our model penalizes not only incorrect answers but also low-confidence correct responses, thereby promoting more robust and logically consistent reasoning. We validate the effectiveness of our approach through static evaluations, Best-of-N inference tests, and PPO-based RL training. Our method outperforms several state-of-the-art open-source reward models across diverse STEM benchmarks. We release our codes and model in https://github.com/qianxiHe147/C2RM.