Process Reward Models for LLM Agents: Practical Framework and Directions

TL;DR

This paper presents AgentPRM, a scalable framework for training LLM agents with process rewards, and introduces InversePRM for learning rewards from demonstrations, demonstrating improved performance on benchmarks.

Contribution

It introduces AgentPRM and InversePRM frameworks, enabling scalable, reward-based training of LLM agents with minimal modifications to existing pipelines.

Findings

Small 3B models outperform GPT-4o baselines when trained with AgentPRM and InversePRM.

AgentPRM requires minimal modifications to RLHF pipelines.

Models trained with these methods show improved performance on ALFWorld benchmark.

Abstract

We introduce Agent Process Reward Models (AgentPRM), a simple and scalable framework for training LLM agents to continually improve through interactions. AgentPRM follows a lightweight actor-critic paradigm, using Monte Carlo rollouts to compute reward targets and optimize policies. It requires minimal modifications to existing RLHF pipelines, making it easy to integrate at scale. Beyond AgentPRM, we propose InversePRM, which learns process rewards directly from demonstrations without explicit outcome supervision. We also explore key challenges and opportunities, including exploration, process reward shaping, and model-predictive reasoning. We evaluate on ALFWorld benchmark, show that small 3B models trained with AgentPRM and InversePRM outperform strong GPT-4o baselines, and analyze test-time scaling, reward hacking, and more. Our code is available at: https://github.com/sanjibanc/agent_prm.