Long-form RewardBench: Evaluating Reward Models for Long-form Generation

TL;DR

This paper introduces Long-form RewardBench, a comprehensive benchmark for evaluating reward models in long-form generation tasks, revealing current models' limitations and differences between classifier and generative approaches.

Contribution

It presents the first dedicated benchmark for long-form reward modeling, including a novel test and extensive evaluation of 20+ reward models across multiple subtasks.

Findings

Current reward models lack effective long-form generation capabilities.

Reward performance correlates with response length and error position.

Classifiers generalize better than generative models.

Abstract

The widespread adoption of reinforcement learning-based alignment highlights the growing importance of reward models. Various benchmarks have been built to evaluate reward models in various domains and scenarios. However, a significant gap remains in assessing reward models for long-form generation, despite its critical role in real-world applications. To bridge this, we introduce Long-form RewardBench, the first reward modeling testbed specifically designed for long-form generation. Our benchmark encompasses five key subtasks: QA, RAG, Chat, Writing, and Reasoning. We collected instruction and preference data through a meticulously designed multi-stage data collection process, and conducted extensive experiments on 20+ mainstream reward models, including both classifiers and generative models. Our findings reveal that current models still lack long-form reward modeling capabilities. Furthermore, we designed a novel Long-form Needle-in-a-Haystack Test, which revealed a correlation between reward modeling performance and the error's position within a response, as well as the overall response length, with distinct characteristics observed between classification and generative models. Finally, we demonstrate that classifiers exhibit better generalizability compared to generative models trained on the same data. As the first benchmark for long-form reward modeling, this work aims to offer a robust platform for visualizing progress in this crucial area.