UFT: Unifying Supervised and Reinforcement Fine-Tuning

TL;DR

UFT introduces a unified post-training method combining supervised and reinforcement fine-tuning, significantly improving reasoning abilities and convergence speed of large language models across different sizes.

Contribution

The paper proposes UFT, a novel unified fine-tuning approach that integrates SFT and RFT, overcoming their individual limitations and accelerating reasoning convergence.

Findings

UFT outperforms SFT and RFT across various model sizes.

UFT breaks the exponential sample complexity bottleneck of RFT.

Theoretically proves exponential acceleration in convergence for long-horizon reasoning.

Abstract

Post-training has demonstrated its importance in enhancing the reasoning capabilities of large language models (LLMs). The primary post-training methods can be categorized into supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT). SFT is efficient and well-suited for small language models, but it may lead to overfitting and limit the reasoning abilities of larger models. In contrast, RFT generally yields better generalization but depends heavily on the strength of the base model. To address the limitations of SFT and RFT, we propose Unified Fine-Tuning (UFT), a novel post-training paradigm that unifies SFT and RFT into a single, integrated process. UFT enables the model to effectively explore solutions while incorporating informative supervision signals, bridging the gap between memorizing and thinking underlying existing methods. Notably, UFT outperforms both SFT and RFT in general, regardless of model sizes. Furthermore, we theoretically prove that UFT breaks RFT's inherent exponential sample complexity bottleneck, showing for the first time that unified training can exponentially accelerate convergence on long-horizon reasoning tasks.