RewardFlow: Topology-Aware Reward Propagation on State Graphs for Agentic RL with Large Language Models

TL;DR

RewardFlow is a novel, topology-aware reward propagation method that improves state-level reward estimation in reinforcement learning for large language models, enhancing agentic reasoning performance.

Contribution

It introduces a lightweight, graph-based reward estimation technique that leverages state topology to improve reinforcement learning in agentic reasoning tasks.

Findings

Outperforms prior RL methods on four benchmarks

Demonstrates superior robustness and efficiency

Provides a scalable approach for state-level reward estimation

Abstract

Reinforcement learning (RL) holds significant promise for enhancing the agentic reasoning capabilities of large language models (LLMs) with external environments. However, the inherent sparsity of terminal rewards hinders fine-grained, state-level optimization. Although process reward modeling offers a promising alternative, training dedicated reward models often entails substantial computational costs and scaling difficulties. To address these challenges, we introduce RewardFlow, a lightweight method for estimating state-level rewards tailored to agentic reasoning tasks. RewardFlow leverages the intrinsic topological structure of states within reasoning trajectories by constructing state graphs. This enables an analysis of state-wise contributions to success, followed by topology-aware graph propagation to quantify contributions and yield objective, state-level rewards. When integrated as dense rewards for RL optimization, RewardFlow substantially outperforms prior RL baselines across four agentic reasoning benchmarks, demonstrating superior performance, robustness, and training efficiency. The implementation of RewardFlow is publicly available at https://github.com/tmlr-group/RewardFlow.