RePO: Bridging On-Policy Learning and Off-Policy Knowledge through Rephrasing Policy Optimization

TL;DR

RePO introduces a novel method that combines on-policy and off-policy learning by rephrasing off-policy knowledge into on-policy style, enhancing hard-sample learning and achieving state-of-the-art results in language model fine-tuning.

Contribution

The paper proposes Rephrasing Policy Optimization (RePO), a new approach that improves off-policy knowledge integration while maintaining on-policy stability in language model training.

Findings

RePO outperforms existing methods on several benchmarks.

RePO enhances hard-sample utilization effectively.

RePO achieves state-of-the-art performance in domain-specific language model fine-tuning.

Abstract

Aligning large language models (LLMs) on domain-specific data remains a fundamental challenge. Supervised fine-tuning (SFT) offers a straightforward way to inject domain knowledge but often degrades the model's generality. In contrast, on-policy reinforcement learning (RL) preserves generality but fails to effectively assimilate hard samples that exceed the model's current reasoning level. Recent off-policy RL attempts improve hard sample utilization, yet they suffer from severe training instability due to the forced distribution shift toward off-policy knowledge. To reconcile effective off-policy knowledge absorption with the stability of on-policy RL, we propose Rephrasing Policy Optimization (RePO). In RePO, the policy model is prompted to first comprehend off-policy knowledge and then rephrase it into trajectories that conform to its own stylistic and parametric distribution. RePO dynamically replaces low-reward rollouts with these rephrased, high-quality trajectories. This strategy guides the model toward correct reasoning paths while strictly preserving on-policy training dynamics. Experiments on several benchmarks demonstrate that RePO improves hard-sample utilization and outperforms existing baselines, achieving state-of-the-art performance.