On-Policy Supervised Fine-Tuning for Efficient Reasoning

TL;DR

This paper introduces a simplified on-policy supervised fine-tuning method for large reasoning models that improves efficiency and maintains accuracy by focusing on correctness and conciseness, outperforming complex reinforcement learning approaches.

Contribution

It demonstrates that removing complex reward components and using on-policy supervised fine-tuning achieves better efficiency and comparable or superior reasoning performance.

Findings

Reduces chain-of-thought length by up to 80% while maintaining accuracy.

Cuts GPU memory usage by 50% and speeds up training convergence by 70%.

Outperforms complex RL-based methods across five benchmarks.

Abstract

Large reasoning models (LRMs) are commonly trained with reinforcement learning (RL) to explore long chain-of-thought reasoning, achieving strong performance at high computational cost. Recent methods add multi-reward objectives to jointly optimize correctness and brevity, but these complex extensions often destabilize training and yield suboptimal trade-offs. We revisit this objective and challenge the necessity of such complexity. Through principled analysis, we identify fundamental misalignments in this paradigm: KL regularization loses its intended role when correctness and length are directly verifiable, and group-wise normalization becomes ambiguous under multiple reward signals. By removing these two items and simplifying the reward to a truncation-based length penalty, we show that the optimization problem reduces to supervised fine-tuning on self-generated data filtered for both correctness and conciseness. We term this simplified training strategy on-policy SFT. Despite its simplicity, on-policy SFT consistently defines the accuracy-efficiency Pareto frontier. It reduces CoT length by up to 80 while maintaining original accuracy, surpassing more complex RL-based methods across five benchmarks. Furthermore, it significantly enhances training efficiency, reducing GPU memory usage by 50% and accelerating convergence by 70%. Our code is available at https://github.com/EIT-NLP/On-Policy-SFT.