AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress

TL;DR

This paper introduces AgentPRM, a process reward model for LLM agents that evaluates decision progress and guides multi-turn tasks, improving efficiency and robustness without extensive prompt engineering.

Contribution

We propose a novel process reward model tailored for agent tasks, capturing decision interdependence and progress, with a scalable TD-based training method that enhances efficiency.

Findings

AgentPRM is over 8 times more compute-efficient than baselines.

AgentPRM shows robust performance improvements when scaling test-time compute.

Applying AgentPRM to reinforcement learning offers additional insights.

Abstract

Despite rapid development, large language models (LLMs) still encounter challenges in multi-turn decision-making tasks (i.e., agent tasks) like web shopping and browser navigation, which require making a sequence of intelligent decisions based on environmental feedback. Previous work for LLM agents typically relies on elaborate prompt engineering or fine-tuning with expert trajectories to improve performance. In this work, we take a different perspective: we explore constructing process reward models (PRMs) to evaluate each decision and guide the agent's decision-making process. Unlike LLM reasoning, where each step is scored based on correctness, actions in agent tasks do not have a clear-cut correctness. Instead, they should be evaluated based on their proximity to the goal and the progress they have made. Building on this insight, we propose a re-defined PRM for agent tasks, named AgentPRM, to capture both the interdependence between sequential decisions and their contribution to the final goal. This enables better progress tracking and exploration-exploitation balance. To scalably obtain labeled data for training AgentPRM, we employ a Temporal Difference-based (TD-based) estimation method combined with Generalized Advantage Estimation (GAE), which proves more sample-efficient than prior methods. Extensive experiments across different agentic tasks show that AgentPRM is over $8\times$ more compute-efficient than baselines, and it demonstrates robust improvement when scaling up test-time compute. Moreover, we perform detailed analyses to show how our method works and offer more insights, e.g., applying AgentPRM to the reinforcement learning of LLM agents.