From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR

TL;DR

This paper systematically analyzes how large language models explore during reinforcement learning with verifiable rewards, providing insights and metrics to improve their reasoning capabilities.

Contribution

It introduces a comprehensive framework for understanding exploration mechanisms in RLVR, including new metrics and empirical analyses of exploration behaviors.

Findings

Development of quantitative metrics for exploration boundaries

Analysis of entropy-performance trade-offs at various stages

Methods to enhance exploration-driven performance improvements

Abstract

Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs). Unlike traditional RL approaches, RLVR leverages rule-based feedback to guide LLMs in generating and refining complex reasoning chains -- a process critically dependent on effective exploration strategies. While prior work has demonstrated RLVR's empirical success, the fundamental mechanisms governing LLMs' exploration behaviors remain underexplored. This technical report presents a systematic investigation of exploration capacities in RLVR, covering four main aspects: (1) exploration space shaping, where we develop quantitative metrics to characterize LLMs' capability boundaries; (2) entropy-performance exchange, analyzed across training stages, individual instances, and token-level patterns; and (3) RL performance optimization, examining methods to effectively translate exploration gains into measurable improvements. By unifying previously identified insights with new empirical evidence, this work aims to provide a foundational framework for advancing RLVR systems.