Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them

TL;DR

This paper compares reinforcement learning and supervised fine-tuning for training large language models on reasoning tasks, revealing different impacts on model capabilities and knowledge retention.

Contribution

It provides a detailed analysis of how RL and SFT differently modify model parameters and affect performance on various benchmarks, offering insights into their distinct training dynamics.

Findings

RL yields minor in-domain gains and slight knowledge degradation

SFT causes more significant parameter updates and out-of-domain performance decline

Freezing model parts during training shows inconclusive effects

Abstract

Training large language models (LLMs) for reasoning via maths and code datasets has become a major new focus in LLM post-training. Two particularly popular approaches are reinforcement learning (RL) and supervised fine-tuning (SFT), but their training dynamics are poorly understood. We present a comparative analysis of RL and SFT on the same maths problems with the same model and similar hyperparameters. We find that RL yields minor in-domain gains on maths and slight degradation on knowledge-intensive benchmarks like MMLU, while both trends are more pronounced in SFT. We also analyse model parameters across checkpoints, observing that both algorithms modify query and key weights the most. Meanwhile, SFT exhibits greater updates and also affects mid-layer MLPs more, leading us to hypothesise that this may have caused the out-of-domain degradation. We therefore investigate whether freezing parts of the model during training can mitigate the reduced performance on knowledge-intensive benchmarks. However, our results are inconclusive, with benefits on GPQA:Diamond and degradation on other benchmarks. Taken together, our observations provide a preliminary indication for why RL amplifies existing capabilities, while SFT replaces old skills with new ones.