RIFT: Repurposing Negative Samples via Reward-Informed Fine-Tuning

TL;DR

RIFT is a novel fine-tuning framework that effectively utilizes all self-generated samples, including negative ones, by reweighting with scalar rewards, improving data efficiency and performance in LLM alignment.

Contribution

RIFT introduces a reward-informed loss reweighting method that repurposes negative samples, overcoming training collapse and enhancing alignment performance.

Findings

RIFT outperforms RFT on mathematical benchmarks.

RIFT demonstrates robustness and data efficiency across models.

The stabilized loss formulation ensures numerical robustness.

Abstract

While Supervised Fine-Tuning (SFT) and Rejection Sampling Fine-Tuning (RFT) are standard for LLM alignment, they either rely on costly expert data or discard valuable negative samples, leading to data inefficiency. To address this, we propose Reward Informed Fine-Tuning (RIFT), a simple yet effective framework that utilizes all self-generated samples. Unlike the hard thresholding of RFT, RIFT repurposes negative trajectories, reweighting the loss with scalar rewards to learn from both the positive and negative trajectories from the model outputs. To overcome the training collapse caused by naive reward integration, where direct multiplication yields an unbounded loss, we introduce a stabilized loss formulation that ensures numerical robustness and optimization efficiency. Extensive experiments on mathematical benchmarks across various base models show that RIFT consistently outperforms RFT. Our results demonstrate that RIFT is a robust and data-efficient alternative for alignment using mixed-quality, self-generated data.