LLMs Can Learn to Reason Via Off-Policy RL

TL;DR

This paper introduces OAPL, a novel off-policy RL algorithm for LLMs that handles policy lag effectively, outperforming existing methods on benchmarks and enabling efficient post-training with significant off-policy data.

Contribution

The paper proposes OAPL, an off-policy RL algorithm that does not require policy alignment modifications, effectively managing large policy lags in training and inference.

Findings

OAPL outperforms GRPO with importance sampling on math benchmarks.

OAPL matches DeepCoder's performance on LiveCodeBench with fewer generations.

Models trained with OAPL show improved test time scaling under Pass@k.

Abstract

Reinforcement learning (RL) approaches for Large Language Models (LLMs) frequently use on-policy algorithms, such as PPO or GRPO. However, policy lag from distributed training architectures and differences between the training and inference policies break this assumption, making the data off-policy by design. To rectify this, prior work has focused on making this off-policy data appear more on-policy, either via importance sampling (IS), or by more closely aligning the training and inference policies by explicitly modifying the inference engine. In this work, we embrace off-policyness and propose a novel off-policy RL algorithm that does not require these modifications: Optimal Advantage-based Policy Optimization with Lagged Inference policy (OAPL). We show that OAPL outperforms GRPO with importance sampling on competition math benchmarks, and can match the performance of a publicly available coding model, DeepCoder, on LiveCodeBench, while using 3x fewer generations during training. We further empirically demonstrate that models trained via OAPL have improved test time scaling under the Pass@k metric. OAPL allows for efficient, effective post-training even with lags of more than 400 gradient steps between the training and inference policies, 100x more off-policy than prior approaches.