RM-Distiller: Exploiting Generative LLM for Reward Model Distillation

TL;DR

This paper introduces RM-Distiller, a novel framework that leverages the multifaceted capabilities of generative LLMs to improve reward model distillation, leading to better alignment with human preferences.

Contribution

RM-Distiller systematically exploits the refinement, scoring, and generation capabilities of teacher LLMs for enhanced reward model distillation, a novel approach in the field.

Findings

Outperforms traditional distillation methods on RM benchmarks

Improves reinforcement learning-based alignment results

Demonstrates the importance of multifaceted teacher capabilities

Abstract

Reward models (RMs) play a pivotal role in aligning large language models (LLMs) with human preferences. Due to the difficulty of obtaining high-quality human preference annotations, distilling preferences from generative LLMs has emerged as a standard practice. However, existing approaches predominantly treat teacher models as simple binary annotators, failing to fully exploit the rich knowledge and capabilities for RM distillation. To address this, we propose RM-Distiller, a framework designed to systematically exploit the multifaceted capabilities of teacher LLMs: (1) Refinement capability, which synthesizes highly correlated response pairs to create fine-grained and contrastive signals. (2) Scoring capability, which guides the RM in capturing precise preference strength via a margin-aware optimization objective. (3) Generation capability, which incorporates the teacher's generative distribution to regularize the RM to preserve its fundamental linguistic knowledge. Extensive experiments demonstrate that RM-Distiller significantly outperforms traditional distillation methods both on RM benchmarks and reinforcement learning-based alignment, proving that exploiting multifaceted teacher capabilities is critical for effective reward modeling. To the best of our knowledge, this is the first systematic research on RM distillation from generative LLMs.