Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models

TL;DR

This paper introduces a multi-dimensional evaluation framework and analysis method for reward models, improving interpretability and alignment by probing preference representations across different preference dimensions.

Contribution

It presents MRMBench, a benchmark with six probing tasks for preference dimensions, and inference-time probing for better interpretability of reward models.

Findings

MRMBench correlates with LLM alignment performance

Reward models often struggle with multi-dimensional preferences

Inference-time probing improves reward prediction confidence

Abstract

Previous methods evaluate reward models by testing them on a fixed pairwise ranking test set, but they typically do not provide performance information on each preference dimension. In this work, we address the evaluation challenge of reward models by probing preference representations. To confirm the effectiveness of this evaluation method, we construct a Multi-dimensional Reward Model Benchmark (MRMBench), a collection of six probing tasks for different preference dimensions. We design it to favor and encourage reward models that better capture preferences across different dimensions. Furthermore, we introduce an analysis method, inference-time probing, which identifies the dimensions used during the reward prediction and enhances its interpretability. Through extensive experiments, we find that MRMBench strongly correlates with the alignment performance of large language models (LLMs), making it a reliable reference for developing advanced reward models. Our analysis of MRMBench evaluation results reveals that reward models often struggle to capture preferences across multiple dimensions, highlighting the potential of multi-objective optimization in reward modeling. Additionally, our findings show that the proposed inference-time probing method offers a reliable metric for assessing the confidence of reward predictions, which ultimately improves the alignment of LLMs.