DORA Explorer: Improving the Exploration Ability of LLMs Without Training

TL;DR

DORA Explorer is a training-free framework that enhances the exploration capabilities of LLM agents in decision-making tasks by generating and scoring diverse actions, leading to improved performance in environments like MAB and TALES.

Contribution

The paper introduces DORA Explorer, a novel, training-free method that significantly improves exploration in LLM agents, outperforming existing decoding strategies in various environments.

Findings

DORA achieves UCB-competitive performance on Multi-Armed Bandit tasks.

DORA improves TextWorld performance from 29.2% to 45.5%.

Existing prompting methods are insufficient for robust exploration.

Abstract

Despite the rapid progress, LLMs for sequential decision-making (i.e., LLM agents) still struggle to produce diverse outputs. This leads to insufficient exploration, convergence to sub-optimal solutions, and becoming stuck in loops. Such limitations can be problematic in environments that require active exploration to gather information and make decisions. Sampling methods such as temperature scaling introduce token-level randomness but fail to produce enough diversity at the sequence level. We analyze LLM exploration in the classic Multi-Armed Bandit (MAB) setting and the Text Adventure Learning Environment Suite (TALES). We find that current decoding strategies and prompting methods like Chain-of-Thought and Tree-of-Thought are insufficient for robust exploration. To address this, we introduce DORA Explorer (Diversity-Oriented Ranking of Actions), a training-free framework for improving exploration in LLM agents. DORA generates diverse action candidates, scores them using token log-probabilities, and selects actions using a tunable exploration parameter. DORA achieves UCB-competitive performance on MAB and consistent gains across TALES, e.g., improving Qwen2.5-7B's performance from 29.2% to 45.5% in TextWorld. Our project is available at: https://dora-explore.github.io/.