Failing to Explore: Language Models on Interactive Tasks

TL;DR

This paper evaluates how well language models explore interactive environments with limited interactions, revealing systematic under-exploration and proposing interventions to improve exploration efficiency.

Contribution

It introduces three controllable exploration tasks and studies lightweight interventions that enhance exploration performance of language models.

Findings

Models under-explore and perform worse than simple heuristics.

Splitting interaction budget into parallel runs improves exploration.

Periodic summarization preserves discoveries and boosts exploration.

Abstract

We evaluate language models on their ability to explore interactive environments under a limited interaction budget. We introduce three parametric tasks with controllable exploration difficulty, spanning continuous and discrete environments. Across state-of-the-art models, we find systematic under-exploration and suboptimal solutions, with performance often significantly worse than simple explore--exploit heuristic baselines and scaling weakly as the budget increases. Finally, we study two lightweight interventions: splitting a fixed budget into parallel executions, which surprisingly improves performance despite a no-gain theoretical result for our tasks, and periodically summarizing the interaction history, which preserves key discoveries and further improves exploration.