R$^2$PO: Decoupling Training Trajectories from Inference Responses for LLM Reasoning

TL;DR

R$^2$PO introduces a method to decouple training trajectories from inference responses in LLMs, enhancing reasoning by allowing controlled exploration during training without affecting inference stability.

Contribution

The paper proposes R$^2$PO, a lightweight module that separates training trajectories from inference responses, improving exploration and reasoning performance in LLMs.

Findings

Achieves 3.4% accuracy gain on MATH-500

Achieves 1.3% accuracy gain on APPS

Reduces formatting errors and length bias

Abstract

Reinforcement learning has become a central paradigm for improving LLM reasoning. However, existing methods use a single policy to produce both inference responses and training optimization trajectories. The objective conflict between generating stable inference responses and diverse training trajectories leads to insufficient exploration, which harms reasoning capability. In this paper, to address the problem, we propose R$^2$PO (Residual Rollout Policy Optimization), which introduces a lightweight Residual Rollout-Head atop the policy to decouple training trajectories from inference responses, enabling controlled trajectory diversification during training while keeping inference generation stable. Experiments across multiple benchmarks show that our method consistently outperforms baselines, achieving average accuracy gains of 3.4% on MATH-500 and 1.3% on APPS, while also reducing formatting errors and mitigating length bias for stable optimization. Our code is publicly available at https://github.com/RRPO-ARR/Code.