Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimization

TL;DR

This paper introduces an exploration-aware reinforcement learning framework for large language models that adaptively explores based on uncertainty, improving decision-making in text and GUI tasks.

Contribution

It presents a novel variational inference-based reward function and grouping mechanism enabling selective exploration in LLM agents.

Findings

Achieves consistent improvements on text-based benchmarks.

Effectively distinguishes when exploration is necessary.

Enhances decision-making by targeting informational gaps.

Abstract

Recent advancements in agentic test-time scaling allow models to gather environmental feedback before committing to final actions. A key limitation of existing methods is that they typically employ undifferentiated exploration strategies, lacking the ability to adaptively distinguish when exploration is truly required. In this paper, we propose an exploration-aware reinforcement learning framework that enables LLM agents to adaptively explore only when uncertainty is high. Our method introduces a fine-grained reward function via variational inference that explicitly evaluates exploratory actions by estimating their potential to improve future decision-making, together with an exploration-aware grouping mechanism that separates exploratory actions from task-completion actions during optimization. By targeting informational gaps, this design allows agents to explore selectively and transition to execution as soon as the task context is clear. Empirically, we demonstrate that our approach achieves consistent improvements across a range of challenging text-based and GUI-based agent benchmarks. Code is available at https://github.com/HansenHua/EAPO-ICML26 and models are available at https://huggingface.co/hansenhua/EAPO-ICML26.