Scaling Reasoning Efficiently via Relaxed On-Policy Distillation

TL;DR

This paper introduces REOPOLD, a novel framework for stabilizing on-policy distillation by relaxing imitation constraints, leading to more efficient training and better reasoning performance in capacity-limited models.

Contribution

REOPOLD provides a new method that stabilizes on-policy distillation through reward relaxation techniques, improving sample efficiency and inference scaling.

Findings

REOPOLD achieves 6.7-12x greater sample efficiency than recent RL methods.

REOPOLD enables a 7B student to match a 32B teacher in visual reasoning.

REOPOLD improves inference speed by approximately 3.32 times.

Abstract

On-policy distillation is pivotal for transferring reasoning capabilities to capacity-constrained models, yet remains prone to instability and negative transfer. We show that on-policy distillation can be interpreted, both theoretically and empirically, as a form of policy optimization, where the teacher-student log-likelihood ratio acts as a token reward. From this insight, we introduce REOPOLD (Relaxed On-Policy Distillation) a framework that stabilizes optimization by relaxing the strict imitation constraints of standard on-policy distillation. Specifically, REOPOLD temperately and selectively leverages rewards from the teacher through mixture-based reward clipping, entropy-based token-level dynamic sampling, and a unified exploration-to-refinement training strategy. Empirically, REOPOLD surpasses its baselines with superior sample efficiency during training and enhanced test-time scaling at inference, across mathematical, visual, and agentic tool-use reasoning tasks. Specifically, REOPOLD outperforms recent RL approaches achieving 6.7~12x greater sample efficiency and enables a 7B student to match a 32B teacher in visual reasoning with a ~3.32x inference speedup.