OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning

TL;DR

OpenReward introduces a tool-augmented reward model that improves evaluation of long-form, knowledge-intensive tasks by incorporating external evidence, leading to better alignment of large language models.

Contribution

We develop OpenRM, a novel reward model that uses external tools for evidence gathering, trained with a new synthesis framework, enhancing long-form response evaluation.

Findings

OpenRM outperforms existing reward models on multiple datasets.

Incorporating external tools improves evaluation accuracy for knowledge-intensive tasks.

OpenRM enhances LLM alignment in response selection and data curation.

Abstract

Reward models (RMs) have become essential for aligning large language models (LLMs), serving as scalable proxies for human evaluation in both training and inference. However, existing RMs struggle on knowledge-intensive and long-form tasks, where evaluating correctness requires grounding beyond the model's internal knowledge. This limitation hinders them from reliably discriminating subtle quality differences, especially when external evidence is necessary. To address this, we introduce OpenRM, a tool-augmented long-form reward model that systematically judges open-ended responses by invoking external tools to gather relevant evidence. We train OpenRM with Group Relative Policy Optimization (GRPO) on over 27K synthesized pairwise examples generated through a controllable data synthesis framework. The training objective jointly supervises intermediate tool usage and final outcome accuracy, incentivizing our reward model to learn effective evidence-based judgment strategies. Extensive experiments on three newly-collected datasets and two widely-used benchmarks demonstrate that OpenRM substantially outperforms existing reward modeling approaches. As a further step, we integrate OpenRM into both inference-time response selection and training-time data selection. This yields consistent gains in downstream LLM alignment tasks, highlighting the potential of tool-augmented reward models for scaling reliable long-form evaluation.