Can Large Reasoning Models Self-Train?

TL;DR

This paper investigates the potential of self-training in large reasoning models using reinforcement learning and majority voting, revealing both its benefits and critical limitations like reward hacking that hinder sustained self-improvement.

Contribution

It demonstrates that simple self-feedback mechanisms can improve reasoning performance but also identifies fundamental challenges such as reward hacking in prolonged self-training.

Findings

Self-training improves reasoning performance.

Better feedback quality enhances model learning.

Reward hacking causes performance collapse.

Abstract

Recent successes of reinforcement learning (RL) in training large reasoning models motivate the question of whether self-training - the process where a model learns from its own judgments - can be sustained within RL. In this work, we study this question using majority voting as a simple self-feedback mechanism. On a comprehensive set of experiments on both synthetic and real reasoning tasks, we find that this basic approach improves not only the model's reasoning performance, but also its capability of generating better quality feedback for the next RL iteration, driving further model improvement. Yet our analysis also reveals a critical limitation of such a self-training paradigm - prolonged RL with self-reward leads to reward hacking where models learn to maximize training (pseudo-)reward, resulting in sudden and complete performance collapse. Together, these results highlight feedback design as the central challenge and call for future research on mechanisms to enable prolonged self-improvement.