Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling

TL;DR

This paper introduces Prefix-RFT, a hybrid fine-tuning method that combines supervised and reinforcement learning techniques to improve large language model performance on reasoning tasks.

Contribution

It presents a novel unified approach that effectively merges supervised and reinforcement fine-tuning, outperforming individual and parallel methods.

Findings

Prefix-RFT surpasses standalone SFT and RFT performance.

It outperforms parallel mixed-policy RFT methods.

The approach is robust to variations in demonstration data quality and quantity.

Abstract

Existing LLMs-post-training techniques are broadly categorized into supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT). Each paradigm presents a distinct trade-off: (1) SFT excels at mimicking demonstration data, but can lead to problematic generalization as a form of behavior cloning. (2) Conversely, RFT can significantly enhance a model's performance but is prone to learning unexpected behaviors, and its performance is sensitive to the initial policy. In this paper, we propose a unified view of these methods and introduce Prefix-RFT, a hybrid approach that synergizes learning from both demonstration and exploration. Using mathematical reasoning problems as a test bed, we empirically demonstrate that Prefix-RFT is simple yet effective. Not only does it surpass the performance of standalone SFT and RFT, but it also outperforms parallel mixed-policy RFT methods. Our analysis highlights the complementary nature of SFT and RFT, validating that Prefix-RFT effectively harmonizes them. Further ablation studies confirm the method's robustness to variations in the quality and quantity of demonstration data.