TextQuests: How Good are LLMs at Text-Based Video Games?

TL;DR

TextQuests introduces a new benchmark based on interactive fiction games to evaluate LLMs' autonomous reasoning and problem-solving abilities in complex, exploratory, text-based environments without external tools.

Contribution

The paper presents TextQuests, a novel benchmark using Infocom games to assess LLMs' long-term reasoning and autonomous problem-solving in challenging interactive environments.

Findings

Benchmark effectively measures LLMs' reasoning in exploratory tasks

TextQuests reveals strengths and limitations of current LLMs in self-contained problem-solving

Provides a new standard for evaluating AI in complex, interactive text environments

Abstract

Evaluating AI agents within complex, interactive environments that mirror real-world challenges is critical for understanding their practical capabilities. While existing agent benchmarks effectively assess skills like tool use or performance on structured tasks, they often do not fully capture an agent's ability to operate autonomously in exploratory environments that demand sustained, self-directed reasoning over a long and growing context. To enable a more accurate assessment of AI agents in challenging exploratory environments, we introduce TextQuests, a benchmark based on the Infocom suite of interactive fiction games. These text-based adventures, which can take human players over 30 hours and require hundreds of precise actions to solve, serve as an effective proxy for evaluating AI agents on focused, stateful tasks. The benchmark is specifically designed to assess an LLM agent's capacity for self-contained problem-solving by precluding the use of external tools, thereby focusing on intrinsic long-context reasoning capabilities in an exploratory environment characterized by the need for trial-and-error learning and sustained problem-solving within a single interactive session. We release TextQuests at https://textquests.ai.