Reassessing the Role of Supervised Fine-Tuning: An Empirical Study in VLM Reasoning

TL;DR

This paper systematically compares supervised fine-tuning and reinforcement learning in vision-language models, revealing that SFT is crucial for weaker models, data efficiency, and transferability, challenging the dominance of RL in reasoning tasks.

Contribution

It provides a controlled empirical analysis showing the importance of supervised fine-tuning alongside reinforcement learning in vision-language model reasoning.

Findings

SFT improves reasoning in weaker models.

SFT achieves comparable performance with less data.

RL rewards can be deceptive and not correlate with reasoning accuracy.

Abstract

Recent advances in vision-language models (VLMs) reasoning have been largely attributed to the rise of reinforcement Learning (RL), which has shifted the community's focus away from the supervised fine-tuning (SFT) paradigm. Many studies suggest that introducing the SFT stage not only fails to improve reasoning ability but may also negatively impact model training. In this study, we revisit this RL-centric belief through a systematic and controlled comparison of SFT and RL on VLM Reasoning. Using identical data sources, we find that the relative effectiveness of SFT and RL is conditional and strongly influenced by model capacity, data scale, and data distribution. Contrary to common assumptions, our findings show that SFT plays a crucial role across several scenarios: (1) Effectiveness for weaker models. SFT more reliably elicits reasoning capabilities in smaller or weaker VLMs. (2) Data efficiency. SFT with only 2K achieves comparable or better reasoning performance to RL with 20K. (3) Cross-modal transferability. SFT demonstrates stronger generalization across modalities. Moreover, we identify a pervasive issue of deceptive rewards, where higher rewards fail to correlate with better reasoning accuracy in RL. These results challenge the prevailing "RL over SFT" narrative. They highlight that the role of SFT may have been underestimated and support a more balanced post-training pipeline in which SFT and RL function as complementary components.