SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution

TL;DR

SPA-RL introduces a stepwise reward attribution framework that decomposes final task rewards into incremental contributions, improving reinforcement learning training for complex, multi-step LLM agents by providing more effective intermediate signals.

Contribution

The paper proposes SPA, a novel reward redistribution method that enhances RL training of LLM agents by decomposing delayed rewards into stepwise contributions using a progress estimator.

Findings

SPA outperforms state-of-the-art methods in success rate and grounding accuracy.

The method provides more effective intermediate rewards for RL training.

Experiments on Webshop, ALFWorld, and VirtualHome validate SPA's effectiveness.

Abstract

Reinforcement learning (RL) holds significant promise for training LLM agents to handle complex, goal-oriented tasks that require multi-step interactions with external environments. However, a critical challenge when applying RL to these agentic tasks arises from delayed rewards: feedback signals are typically available only after the entire task is completed. This makes it non-trivial to assign delayed rewards to earlier actions, providing insufficient guidance regarding environmental constraints and hindering agent training. In this work, we draw on the insight that the ultimate completion of a task emerges from the cumulative progress an agent makes across individual steps. We propose Stepwise Progress Attribution (SPA), a general reward redistribution framework that decomposes the final reward into stepwise contributions, each reflecting its incremental progress toward overall task completion. To achieve this, we train a progress estimator that accumulates stepwise contributions over a trajectory to match the task completion. During policy optimization, we combine the estimated per-step contribution with a grounding signal for actions executed in the environment as the fine-grained, intermediate reward for effective agent training. Extensive experiments on common agent benchmarks (including Webshop, ALFWorld, and VirtualHome) demonstrate that SPA consistently outperforms the state-of-the-art method in both success rate (+2.5\% on average) and grounding accuracy (+1.9\% on average). Further analyses demonstrate that our method remarkably provides more effective intermediate rewards for RL training. Our code is available at https://github.com/WangHanLinHenry/SPA-RL-Agent.