TextAtari: 100K Frames Game Playing with Language Agents

TL;DR

TextAtari introduces a challenging benchmark translating Atari game states into text to evaluate language agents on long-horizon decision tasks, revealing significant gaps compared to human performance.

Contribution

The paper presents TextAtari, a novel textual benchmark for evaluating language models on extended decision-making tasks, bridging Atari game states with natural language processing.

Findings

Language agents underperform humans in long-term planning tasks.

Different agent frameworks and scenarios significantly affect performance.

Current models face challenges in sequential reasoning and strategic planning.

Abstract

We present TextAtari, a benchmark for evaluating language agents on very long-horizon decision-making tasks spanning up to 100,000 steps. By translating the visual state representations of classic Atari games into rich textual descriptions, TextAtari creates a challenging test bed that bridges sequential decision-making with natural language processing. The benchmark includes nearly 100 distinct tasks with varying complexity, action spaces, and planning horizons, all rendered as text through an unsupervised representation learning framework (AtariARI). We evaluate three open-source large language models (Qwen2.5-7B, Gemma-7B, and Llama3.1-8B) across three agent frameworks (zero-shot, few-shot chain-of-thought, and reflection reasoning) to assess how different forms of prior knowledge affect performance on these long-horizon challenges. Four scenarios-Basic, Obscured, Manual Augmentation, and Reference-based-investigate the impact of semantic understanding, instruction comprehension, and expert demonstrations on agent decision-making. Our results reveal significant performance gaps between language agents and human players in extensive planning tasks, highlighting challenges in sequential reasoning, state tracking, and strategic planning across tens of thousands of steps. TextAtari provides standardized evaluation protocols, baseline implementations, and a framework for advancing research at the intersection of language models and planning. Our code is available at https://github.com/Lww007/Text-Atari-Agents.