From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning

TL;DR

This paper introduces StepAgent, a reinforcement learning approach for LLM agents that uses step-wise rewards and expert comparisons to improve policy learning and performance in complex tasks.

Contribution

The paper proposes a novel step-wise reward mechanism and inverse reinforcement learning techniques to enhance LLM agent training, addressing sparse reward issues.

Findings

StepAgent outperforms baseline methods across datasets.

Action distribution converges to expert actions over training.

Intermediate rewards improve policy learning efficiency.

Abstract

The outstanding capabilities of large language models (LLMs) render them a crucial component in various autonomous agent systems. While traditional methods depend on the inherent knowledge of LLMs without fine-tuning, more recent approaches have shifted toward the reinforcement learning strategy to further enhance agents' ability to solve complex interactive tasks with environments and tools. However, previous approaches are constrained by the sparse reward issue, where existing datasets solely provide a final scalar reward for each multi-step reasoning chain, potentially leading to ineffectiveness and inefficiency in policy learning. In this paper, we introduce StepAgent, which utilizes step-wise reward to optimize the agent's reinforcement learning process. Inheriting the spirit of novice-to-expert theory, we first compare the actions of the expert and the agent to automatically generate intermediate rewards for fine-grained optimization. Additionally, we propose implicit-reward and inverse reinforcement learning techniques to facilitate agent reflection and policy adjustment. Further theoretical analysis demonstrates that the action distribution of the agent can converge toward the expert action distribution over multiple training cycles. Experimental results across various datasets indicate that StepAgent outperforms existing baseline methods.