MASPRM: Multi-Agent System Process Reward Model

TL;DR

MASPRM is a novel reward model for multi-agent systems that guides inference by assigning values to partial transcripts, improving search efficiency and performance across multiple benchmarks without requiring step-level annotations.

Contribution

It introduces MASPRM, a reward model trained from terminal rewards to guide inference in multi-agent systems, enhancing search and decision quality without human step annotations.

Findings

MASPRM improves Hit@1 by up to 13.4 points across benchmarks.

It reduces the gap between Hit@1 and Hit@5 by up to 10.3 points.

MASPRM effectively guides search in multi-agent inference tasks.

Abstract

Practical deployment of multi-agent systems (MAS) demands strong performance at test time, motivating methods that guide search during inference and selectively spend compute to improve quality. We present the Multi-Agent System Process Reward Model (MASPRM). It assigns values to partial inter-agent transcripts for each action and each agent, and acts as a controller during inference. MASPRM is trained from multi-agent Monte Carlo Tree Search (MCTS) rollouts labeled only with terminal outcome rewards, without requiring human step-level annotations, by propagating returns to local targets. During inference, MASPRM guides step-level beam search (SBS) and MCTS, focusing computation on promising branches and pruning unpromising ones. We train and test MASPRM across different tasks and domains, using GSM8K, MATH, MMLU, and LogiQA as benchmarks. Averaged across these benchmarks, MASPRM improves Hit@1 over policy likelihood by up to $+13.4$ points and improves ranking quality, reducing Hit@1$->$Hit@5 gaps by up to $10.3$ points. MASPRM complements inference-time search by scoring intermediate routed transcripts to guide rollouts in MAS with fixed schedules. Code: https://github.com/milad1378yz/MASPRM