Step-wise Adaptive Integration of Supervised Fine-tuning and Reinforcement Learning for Task-Specific LLMs

TL;DR

This paper introduces SASR, a step-wise adaptive hybrid training framework that unifies supervised fine-tuning and reinforcement learning to improve task-specific large language models, overcoming limitations of static methods.

Contribution

SASR dynamically balances SFT and RL during training using gradient-based adjustments, inspired by curriculum learning, to enhance reasoning abilities of LLMs.

Findings

SASR outperforms SFT, RL, and static hybrid methods in experiments.

The adaptive framework maintains core reasoning skills while exploring new paths.

Dynamic adjustment improves training stability and generalization.

Abstract

Large language models (LLMs) excel at mathematical reasoning and logical problem-solving. The current popular training paradigms primarily use supervised fine-tuning (SFT) and reinforcement learning (RL) to enhance the models' reasoning abilities. However, when using SFT or RL alone, there are respective challenges: SFT may suffer from overfitting, while RL is prone to mode collapse. The state-of-the-art methods have proposed hybrid training schemes. However, static switching faces challenges such as poor generalization across different tasks and high dependence on data quality. In response to these challenges, inspired by the curriculum learning-quiz mechanism in human reasoning cultivation, We propose SASR, a step-wise adaptive hybrid training framework that theoretically unifies SFT and RL and dynamically balances the two throughout optimization. SASR uses SFT for initial warm-up to establish basic reasoning skills, and then uses an adaptive dynamic adjustment algorithm based on gradient norm and divergence relative to the original distribution to seamlessly integrate SFT with the online RL method GRPO. By monitoring the training status of LLMs and adjusting the training process in sequence, SASR ensures a smooth transition between training schemes, maintaining core reasoning abilities while exploring different paths. Experimental results demonstrate that SASR outperforms SFT, RL, and static hybrid training methods.