Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs

TL;DR

This paper investigates how Reinforcement Learning with Verifiable Rewards (RLVR) enhances reasoning in large language models, demonstrating that RLVR encourages correct reasoning and extends reasoning capabilities through a new evaluation metric and theoretical analysis.

Contribution

It introduces the CoT-Pass@K metric and a theoretical framework explaining RLVR's incentive mechanism for improving reasoning in LLMs.

Findings

RLVR extends reasoning boundaries in mathematical and coding tasks.

The new CoT-Pass@K metric effectively measures reasoning success.

RLVR incentivizes correct reasoning early in training.

Abstract

Recent advancements in long chain-of-thought (CoT) reasoning, particularly through the Group Relative Policy Optimization algorithm used by DeepSeek-R1, have led to significant interest in the potential of Reinforcement Learning with Verifiable Rewards (RLVR) for Large Language Models (LLMs). While RLVR promises to improve reasoning by allowing models to learn from free exploration, there remains debate over whether it truly enhances reasoning abilities or simply boosts sampling efficiency. This paper systematically investigates the impact of RLVR on LLM reasoning. We revisit Pass@K experiments and demonstrate that RLVR can extend the reasoning boundary for both mathematical and coding tasks. This is supported by our introduction of a novel evaluation metric, CoT-Pass@K, which captures reasoning success by accounting for both the final answer and intermediate reasoning steps. Furthermore, we present a theoretical framework explaining RLVR's incentive mechanism, demonstrating how it can encourage correct reasoning even when rewards are based solely on answer correctness. Our analysis of RLVR's training dynamics reveals that it incentivizes correct reasoning early in the process, with substantial improvements in reasoning quality confirmed through extensive evaluations. These findings provide strong evidence of RLVR's potential to enhance LLM reasoning, offering valuable insights into its mechanisms and performance improvements.