Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward

TL;DR

This paper investigates the paradoxical effects of spurious rewards and entropy minimization in reinforcement learning with verifiable rewards, revealing how clipping bias and reward misalignment influence model confidence and reasoning performance.

Contribution

It uncovers the role of clipping bias in reducing policy entropy and proposes a reward-misalignment model to explain spurious rewards' benefits in RLVR.

Findings

Clipping bias reduces policy entropy, leading to more deterministic outputs.

Entropy minimization alone does not improve reasoning performance.

Spurious rewards can enhance performance through reward misalignment and model contamination.

Abstract

This paper examines the exploration-exploitation trade-off in reinforcement learning with verifiable rewards (RLVR), a framework for improving the reasoning of Large Language Models (LLMs). Recent studies suggest that RLVR can elicit strong mathematical reasoning in LLMs through two seemingly paradoxical mechanisms: spurious rewards, which suppress exploitation by rewarding outcomes unrelated to the ground truth, and entropy minimization, which suppresses exploration by pushing the model toward more confident and deterministic outputs, highlighting a puzzling dynamic: both discouraging exploitation and discouraging exploration improve reasoning performance, yet the underlying principles that reconcile these effects remain poorly understood. We focus on two fundamental questions: (i) how policy entropy relates to performance, and (ii) whether spurious rewards yield gains, potentially through the interplay of clipping bias and model contamination. Our results show that clipping bias under spurious rewards reduces policy entropy, leading to more confident and deterministic outputs, while entropy minimization alone is insufficient for improvement. We further propose a reward-misalignment model explaining why spurious rewards can enhance performance beyond contaminated settings. Our findings clarify the mechanisms behind spurious-reward benefits and provide principles for more effective RLVR training.