DogeRM: Equipping Reward Models with Domain Knowledge through Model Merging

TL;DR

DogeRM is a framework that integrates domain-specific knowledge into reward models via model merging, improving alignment efficiency and performance in reinforcement learning from human feedback.

Contribution

The paper introduces DogeRM, a novel method for incorporating domain knowledge into reward models through model merging, reducing data collection costs and enhancing model alignment.

Findings

Improved performance across multiple benchmarks.

Effective integration of domain knowledge into reward models.

Potential to reduce annotation costs in RLHF.

Abstract

Reinforcement learning from human feedback (RLHF) is a popular strategy for aligning large language models (LLMs) with desired behaviors. Reward modeling is a crucial step in RLHF. However, collecting paired preference data for training reward models is often costly and time-consuming, especially for domain-specific preferences requiring expert annotation. To address this challenge, we propose the \textbf{Do}main knowled\textbf{ge} merged \textbf{R}eward \textbf{M}odel (DogeRM), a novel framework that integrates domain-specific knowledge into a general reward model by model merging. The experiments demonstrate that DogeRM enhances performance across different benchmarks and provide a detailed analysis showcasing the effects of model merging, showing the great potential of facilitating model alignment.