Harnessing Negative Signals: Reinforcement Distillation from Teacher Data for LLM Reasoning

TL;DR

This paper introduces a novel reinforcement distillation method that leverages both correct and incorrect reasoning traces to improve large language model reasoning, achieving high performance with significantly less data.

Contribution

It proposes the REDI objective and a two-stage training process that effectively utilizes negative reasoning traces, outperforming existing preference optimization methods.

Findings

Qwen-REDI-1.5B achieves 83.1% on MATH-500 with only 131k traces.

The approach matches the performance of models trained on much larger proprietary datasets.

Utilizing negative traces significantly improves data efficiency in LLM distillation.

Abstract

Recent advances in model distillation show that data from advanced reasoning models can effectively train smaller student models. However, standard practices discard incorrect reasoning traces -- valuable, yet underutilized data. This paper addresses the critical question: How can both positive and negative distilled reasoning traces be effectively leveraged to maximize LLM reasoning performance in an offline setting? We employ a two-stage training recipe: first, Supervised Fine-Tuning (SFT) on positive traces, followed by a refinement stage using both positive and negative traces. We find that a simple REINFORCE-style objective, which we term the Reinforcement Distillation (REDI) objective, outperforms established preference optimization methods like DPO and SimPO in this distillation context. Our empirical evaluations demonstrate the effectiveness of this approach. Notably, our Qwen-REDI-1.5B model, trained on just 131k traces from the open Open-R1 dataset, achieves an 83.1% score on MATH-500. Its performance matches that of DeepSeek-R1-Distill-Qwen-1.5B, a model trained on 800k proprietary data. This result showcases the remarkable data efficiency of utilizing previously discarded negative traces.