Multi-Agent Collaborative Reward Design for Enhancing Reasoning in Reinforcement Learning

TL;DR

This paper introduces CRM, a multi-agent framework for reward modeling in reinforcement learning that improves robustness and interpretability by decomposing evaluation into specialized agents and aggregating their signals.

Contribution

The paper proposes CRM, a novel multi-agent collaborative reward model that enhances transparency and stability in RLHF by decomposing preferences and integrating signals from domain-specific evaluators.

Findings

CRM improves robustness of reward signals.

Enhanced interpretability through decomposed evaluation.

Stable policy optimization with multi-perspective rewards.

Abstract

We present CRM (Multi-Agent Collaborative Reward Model), a framework that replaces a single black-box reward model with a coordinated team of specialist evaluators to improve robustness and interpretability in RLHF. Conventional reward models struggle to jointly optimize multiple, sometimes conflicting, preference dimensions (e.g., factuality, helpfulness, safety) and offer limited transparency into why a score is assigned. CRM addresses these issues by decomposing preference evaluation into domain-specific agents that each produce partial signals, alongside global evaluators such as ranker-based and embedding-similarity rewards. A centralized aggregator fuses these signals at each timestep, balancing factors like step-wise correctness, multi-agent agreement, and repetition penalties, yielding a single training reward compatible with standard RL pipelines. The policy is optimized with advantage-based updates (e.g., GAE), while a value model regresses to the aggregated reward, enabling multi-perspective reward shaping without requiring additional human annotations beyond those used to train the evaluators. To support training and assessment, we introduce rewardBench, a benchmark and training suite aligned with the collaborative structure of CRM. Together, CRM and rewardBench provide a practical, modular path to more transparent reward modeling and more stable optimization.