Expanding LLM Agent Boundaries with Strategy-Guided Exploration

TL;DR

This paper introduces Strategy-Guided Exploration (SGE), a novel method that enhances reinforcement learning for large language model agents by guiding exploration through high-level language strategies, leading to improved efficiency and performance.

Contribution

The work proposes SGE, a strategy-based exploration method that shifts focus from low-level actions to high-level language strategies, with mechanisms for diversity and reflection, outperforming existing RL approaches.

Findings

SGE improves learning efficiency across multiple environments.

SGE enables solving tasks beyond base model capabilities.

SGE achieves higher final performance than exploration-focused RL baselines.

Abstract

Reinforcement learning (RL) has demonstrated notable success in post-training large language models (LLMs) as agents for tasks such as computer use, tool calling, and coding. However, exploration remains a central challenge in RL for LLM agents, especially as they operate in language-action spaces with complex observations and sparse outcome rewards. In this work, we address exploration for LLM agents by leveraging the ability of LLMs to plan and reason in language about the environment to shift exploration from low-level actions to higher-level language strategies. We thus propose Strategy-Guided Exploration (SGE), which first generates a concise natural-language strategy that describes what to do to make progress toward the goal, and then generates environment actions conditioned on that strategy. By exploring in the space of strategies rather than the space of actions, SGE induces structured and diverse exploration that targets different environment outcomes. To increase strategy diversity during RL, SGE introduces mixed-temperature sampling, which explores diverse strategies in parallel, along with a strategy reflection process that grounds strategy generation on the outcomes of previous strategies in the environment. Across UI interaction, tool-calling, coding, and embodied agent environments, SGE consistently outperforms exploration-focused RL baselines, improving both learning efficiency and final performance. We show that SGE enables the agent to learn to solve tasks too difficult for the base model.