Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimization

TL;DR

This paper investigates how to better evaluate reward models in reinforcement learning from human feedback by considering reward overoptimization, proposing improved benchmark design principles to more accurately reflect true model capabilities.

Contribution

It introduces new evaluation strategies that account for reward overoptimization, improving the reliability of reward model assessments in RLHF.

Findings

Minimize differences between chosen and rejected responses beyond correctness

Use multiple comparisons across diverse responses

Source responses from various models for evaluation

Abstract

Reward models (RMs) play a crucial role in reinforcement learning from human feedback (RLHF), aligning model behavior with human preferences. However, existing benchmarks for reward models show a weak correlation with the performance of optimized policies, suggesting that they fail to accurately assess the true capabilities of RMs. To bridge this gap, we explore several evaluation designs through the lens of reward overoptimization\textemdash a phenomenon that captures both how well the reward model aligns with human preferences and the dynamics of the learning signal it provides to the policy. The results highlight three key findings on how to construct a reliable benchmark: (i) it is important to minimize differences between chosen and rejected responses beyond correctness, (ii) evaluating reward models requires multiple comparisons across a wide range of chosen and rejected responses, and (iii) given that reward models encounter responses with diverse representations, responses should be sourced from a variety of models. However, we also observe that a extremely high correlation with degree of overoptimization leads to comparatively lower correlation with certain downstream performance. Thus, when designing a benchmark, it is desirable to use the degree of overoptimization as a useful tool, rather than the end goal.