SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning

TL;DR

This paper identifies and corrects critical bugs in baseline implementations, demonstrating that the standard SFT-then-RL approach outperforms recent mixed-policy methods in LLM reasoning tasks.

Contribution

The authors reveal and fix two bugs in existing baselines, showing that the corrected SFT-then-RL pipeline surpasses recent mixed-policy methods in performance.

Findings

Corrected baseline improves math benchmark scores by +3.8 and +22.2 points.

A truncated SFT-then-RL variant outperforms mixed-policy methods with fewer FLOPs.

Faulty baselines underestimated the effectiveness of the standard SFT-then-RL pipeline.

Abstract

Recent mixed-policy optimization methods for LLM reasoning that interleave or blend supervised and reinforcement learning signals report improvements over the standard SFT-then-RL pipeline. We show that numerous recently published research papers rely on a faulty baseline caused by two distinct bugs: a CPU-offloaded optimizer bug in DeepSpeed that silently drops intermediate micro-batches during gradient accumulation (affecting multiple downstream frameworks including TRL, OpenRLHF and Llama-Factory), and a loss aggregation bug in OpenRLHF that incorrectly weights per-mini-batch losses. Together they suppress SFT performance, with the optimizer bug accounting for most of the gap and the loss aggregation bug contributing a smaller additional effect. Once corrected, the standard SFT-then-RL pipeline surpasses every published mixed-policy method we evaluate by +3.8 points on math benchmarks with Qwen2.5-Math-7B and by +22.2 points with Llama-3.1-8B. Even a truncated variant with just 50 RL steps outperforms mixed-policy methods on math benchmarks while using fewer FLOPs.