NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoning

TL;DR

This paper introduces Negative-aware Fine-Tuning (NFT), a supervised learning method that enables large language models to self-improve in math reasoning tasks by leveraging negative feedback, bridging the gap between supervised and reinforcement learning approaches.

Contribution

NFT is a novel supervised approach that models negative feedback implicitly, allowing LLMs to reflect on failures and improve without external teachers, matching RL performance.

Findings

NFT improves math reasoning performance over standard supervised methods.

NFT matches or surpasses RL algorithms like GRPO and DAPO.

NFT and RL methods are theoretically equivalent in strict-on-policy training.

Abstract

Reinforcement Learning (RL) has played a central role in the recent surge of LLMs' math abilities by enabling self-improvement through binary verifier signals. In contrast, Supervised Learning (SL) is rarely considered for such verification-driven training, largely due to its heavy reliance on reference answers and inability to reflect on mistakes. In this work, we challenge the prevailing notion that self-improvement is exclusive to RL and propose Negative-aware Fine-Tuning (NFT) -- a supervised approach that enables LLMs to reflect on their failures and improve autonomously with no external teachers. In online training, instead of throwing away self-generated negative answers, NFT constructs an implicit negative policy to model them. This implicit policy is parameterized with the same positive LLM we target to optimize on positive data, enabling direct policy optimization on all LLMs' generations. We conduct experiments on 7B and 32B models in math reasoning tasks. Results consistently show that through the additional leverage of negative feedback, NFT significantly improves over SL baselines like Rejection sampling Fine-Tuning, matching or even surpassing leading RL algorithms like GRPO and DAPO. Furthermore, we demonstrate that NFT and GRPO are actually equivalent in strict-on-policy training, even though they originate from entirely different theoretical foundations. Our experiments and theoretical findings bridge the gap between SL and RL methods in binary-feedback learning systems.