Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones?

TL;DR

This paper introduces a novel distillation method that transfers both data and reward signals from large language models, enabling smaller models to outperform larger ones through self-supervised reward learning and reinforcement learning.

Contribution

The paper proposes a new distillation pipeline that incorporates reward transfer and self-supervised reward learning, improving the performance of smaller language models beyond their teachers.

Findings

Smaller models surpass larger teacher models in benchmarks.

Self-supervised reward transfer enhances model performance.

The method reduces reliance on external reward supervision.

Abstract

Distilling large language models (LLMs) typically involves transferring the teacher model's responses through supervised fine-tuning (SFT). However, this approach neglects the potential to distill both data (output content) and reward signals (quality evaluations). Extracting reliable reward signals directly from teacher models is challenging, as LLMs are optimized for generation rather than evaluation, often resulting in biased or inconsistent assessments. To address this limitation, we propose a novel distillation pipeline that transfers both responses and rewards. Our method generates pseudo-rewards through a self-supervised mechanism that leverages the inherent structure of both teacher and student responses, enabling reward learning without explicit external evaluation. The reward model subsequently guides reinforcement learning (RL), allowing iterative refinement of the student model after an SFT warm-up phase. Experiments on GSM8K and MMLU-PRO demonstrate that our method consistently outperforms traditional SFT-based approaches, enabling student models to surpass the performance of their teachers. This work highlights the potential for scalable, efficient distillation through structured self-supervised reward learning, reducing dependence on external reward supervision.