$\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving

TL;DR

Re$^2$ enables large language models to improve reasoning by learning to abandon unproductive paths and restart, significantly increasing redo behavior and boosting performance without supervised fine-tuning.

Contribution

Introducing Re$^2$, a reinforcement learning method that allows LLMs to selectively restart reasoning paths, enhancing reasoning efficiency and accuracy without prior supervised training.

Findings

Redo behavior increased from 0.5% to over 30%.

Performance gains over standard RLVR with same compute.

Notable improvements with increased test samples.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has shown promise in enhancing the reasoning performance of large language models (LLMs) by increasing test-time compute. However, even after extensive RLVR training, such models still tend to generate unnecessary and low-quality steps in their chain-of-thought (CoT), leading to inefficient overthinking and lower answer quality. We show that when the initial direction or quality of the CoT is suboptimal, the model often fails to reach the correct answer, even after generating several times more tokens than when the initial CoT is well-initialized. To this end, we introduce Reinforcement Learning with Re-solving (Re$^2$), in which LLMs learn to flexibly abandon unproductive reasoning paths and restart the solution process when necessary, rather than always committing to a final answer. Re$^2$ applies pure reinforcement learning without any preliminary supervised fine-tuning, successfully amplifying the rare redo behavior in vanilla models from only 0.5% to over 30%. This leads to substantial performance gains over standard RLVR under the same training compute budget, and also demonstrates notable improvements in test-time performance as the number of samples increases.