Disentangling Exploration of Large Language Models by Optimal Exploitation

TL;DR

This paper investigates the exploration capabilities of large language models in unknown environments, proposing a decomposition of rewards to better evaluate exploration versus exploitation and revealing insights into their reasoning and exploration behaviors.

Contribution

It introduces a reward decomposition framework based on optimal returns to assess exploration in large language models, highlighting the importance of exploration for reasoning.

Findings

Most models struggle with effective exploration.

Weak exploration correlates with limited reasoning skills.

Decomposition offers insights into behavior differences due to prompt engineering.

Abstract

Exploration is a crucial skill for in-context reinforcement learning in unknown environments. However, it remains unclear if large language models can effectively explore a partially hidden state space. This work isolates exploration as the sole objective, tasking an agent with gathering information that enhances future returns. Within this framework, we argue that measuring agent returns is not sufficient for a fair evaluation. Hence, we decompose missing rewards into their exploration and exploitation components based on the optimal achievable return. Experiments with various models reveal that most struggle to explore the state space, and weak exploration is insufficient. Nevertheless, we found a positive correlation between exploration performance and reasoning capabilities. Our decomposition can provide insights into differences in behaviors driven by prompt engineering, offering a valuable tool for refining performance in exploratory tasks.