DPEPO: Diverse Parallel Exploration Policy Optimization for LLM-based Agents

TL;DR

DPEPO introduces a novel RL algorithm for LLM agents that enables diverse, parallel exploration across multiple environments, significantly improving success rates in complex tasks.

Contribution

The paper proposes DPEPO, a reinforcement learning method that encourages diverse parallel exploration in LLM agents, enhancing environmental understanding and performance.

Findings

DPEPO achieves state-of-the-art success rates on ALFWorld and ScienceWorld.

It maintains efficiency comparable to strong sequential baselines.

The hierarchical reward scheme effectively promotes exploration diversity.

Abstract

Large language model (LLM) agents that follow the sequential "reason-then-act" paradigm have achieved superior performance in many complex tasks.However, these methods suffer from limited exploration and incomplete environmental understanding, as they interact with only a single environment per step. In this paper, we first introduce a novel paradigm that enables an agent to interact with multiple environments simultaneously and share cross-trajectory experiences. Building upon this paradigm, we further propose DPEPO, a reinforcement learning (RL) algorithm that encourages the agent to perform diverse parallel exploration. There are two stages in DPEPO: initial supervised fine-tuning (SFT) imparts basic parallel reasoning and action generation, followed by reinforcement learning stage with a hierarchical reward scheme. We design a parallel trajectory-level success reward and two step-level rewards: Diverse Action Reward and Diverse State Transition Reward, which actively penalize behavioral redundancy and promote broad exploration. Extensive experiments on ALFWorld and ScienceWorld show that DPEPO achieves state-of-the-art (SOTA) success rates, while maintaining comparable efficiency to strong sequential baselines. (Code is available at https://github.com/LePanda026/Code-for-DPEPO)