On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification

TL;DR

This paper introduces Dynamic Fine-Tuning ( extbackslash model), a simple modification to Supervised Fine-Tuning that improves large language models' generalization by stabilizing gradient updates, bridging the gap with reinforcement learning.

Contribution

It provides a theoretically motivated method to enhance SFT's generalization, outperforming standard SFT on various benchmarks with a single-line change.

Findings

extbackslash model outperforms standard SFT on multiple benchmarks.

It improves generalization in math reasoning, code generation, and multi-modal tasks.

Achieves competitive results in offline reinforcement learning.

Abstract

In this work, we present a simple yet theoretically motivated improvement to Supervised Fine-Tuning (SFT) for the Large Language Model (LLM), addressing its limited generalization compared to reinforcement learning (RL). Through mathematical analysis, we reveal that standard SFT gradients implicitly encode a problematic reward structure that may severely restrict the generalization capabilities of model compared to RL. To rectify this, we propose Dynamic Fine-Tuning (\model), stabilizing gradient updates for each token by dynamically rescaling the objective function with the probability of this token. With just a single-line change, the method outperforms standard SFT on multiple difficult benchmarks and base models, from math reasoning to code generation and multi-modal tasks, demonstrating improved generalization. Additionally, \model~achieves competitive results in offline RL settings, providing an effective yet streamlined alternative. By bridging theoretical insights with practical solutions, this work advances the state of SFT. The source code will be available at https://github.com/yongliang-wu/DFT.