PACR: Progressively Ascending Confidence Reward for LLM Reasoning

TL;DR

This paper introduces PACR, a dense reward mechanism for LLM reasoning that guides the model's belief trajectory, improving exploration efficiency and performance in reinforcement learning settings.

Contribution

PACR is a novel, model-intrinsic reward that encodes ascending confidence in the correct answer, enhancing reinforcement learning for LLM reasoning tasks.

Findings

PACR accelerates exploration and reaches reward saturation faster.

PACR improves performance on multiple reasoning benchmarks.

Theoretical analysis confirms the inductive bias constrains exploration to sound reasoning paths.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly improved LLM reasoning, but its sparse, outcome-based reward provides no guidance for intermediate steps, slowing exploration. We propose Progressively Ascending Confidence Reward (PACR), a dense, model-intrinsic reward computed directly from the model's evolving belief in the correct answer. PACR encodes the inductive bias that, along a well-formed reasoning trajectory, the probability of the ground-truth answer should have a generally ascending trend. We provide empirical and theoretical analysis validating that such an inductive bias constrains the exploration search space to regions richer in logically sound reasoning. We demonstrate that PACR accelerates exploration, reaches reward saturation with fewer trajectories, and yields improvements on multiple benchmarks. Our results suggest that dense, model-intrinsic shaping signals can make RLVR training more effective and reliable.