Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)

TL;DR

This paper reveals that supervised fine-tuning on curated data is fundamentally similar to reinforcement learning and introduces an importance weighted variant that improves performance and aligns more closely with RL objectives.

Contribution

It clarifies the connection between supervised fine-tuning and reinforcement learning, and proposes a simple, effective importance weighted method that enhances fine-tuning outcomes.

Findings

Importance weighted SFT outperforms standard SFT.

The method is easy to implement and generalizes to quality scored data.

Achieves 66.7% on AIME 2024 dataset.

Abstract

Behavior Cloning (BC) on curated (or filtered) data is the predominant paradigm for supervised fine-tuning (SFT) of large language models; as well as for imitation learning of control policies. Here, we draw on a connection between this successful strategy and the theory and practice of finding optimal policies via Reinforcement Learning (RL). Building on existing literature, we clarify that SFT can be understood as maximizing a lower bound on the RL objective in a sparse reward setting. Giving support to its often observed good performance. From this viewpoint, we realize that a small modification to SFT leads to an importance weighted variant that behaves closer to training with RL as it: i) optimizes a tighter bound to the RL objective and, ii) can improve performance compared to SFT on curated data. We refer to this variant as importance weighted supervised fine-tuning (iw-SFT). We show that it is easy to implement and can be further generalized to training with quality scored data. The resulting SFT variants are competitive with more advanced RL algorithms for large language models and for training policies in continuous control tasks. For example achieving 66.7% on the AIME 2024 dataset.