RLSR: Reinforcement Learning from Self Reward

TL;DR

This paper introduces a method where large language models self-judge their solutions to generate reward signals, enabling reinforcement learning without external ground truth, thus broadening the applicability of autonomous self-improvement in AI.

Contribution

The authors demonstrate that LLMs can self-judge solutions to provide effective rewards, eliminating the need for external verification and enabling reinforcement learning in complex, reward-scarce domains.

Findings

Models can reliably self-judge without ground truth.

Self-judging enables performance comparable to formal verification.

Self-supervised training led to competitive results in integration tasks.

Abstract

Large language models can generate solutions to complex problems, but training them with reinforcement learning typically requires verifiable rewards that are expensive to create and not possible for all domains. We demonstrate that LLMs can effectively self-improve through self-judging without reference solutions, leveraging the inherent asymmetry between generating and verifying solutions. Our experiments show that models can provide reliable reward signals without ground truth answers, enabling reinforcement learning in domains where verifiable rewards are impractical. By implementing self-judging across Countdown puzzles and integration problems, we achieve performance comparable to formal verification without ground truth solutions. Most notably, Qwen 2.5 7B DeepSeek Distilled trained with self-rewards qualifies for the prestigious MIT Integration Bee competition, performance through self-supervised improvement. When combined with synthetic question generation, we establish a complete self-improvement loop where models generate practice problems, solve them, and evaluate their own performance without any external validation. Our findings demonstrate that LLM judges can provide effective reward signals for training, unlocking reinforcement learning in countless domains previously limited by reward engineering challenges. This work represents a significant step toward autonomous AI systems that continuously improve through self-directed learning rather than human-guided training, potentially accelerating progress across domains where training data is scarce or evaluation is complex.