General Modular Harness for LLM Agents in Multi-Turn Gaming Environments

TL;DR

This paper presents a modular framework for LLM and VLM agents in multi-turn gaming environments, enabling flexible composition of perception, memory, and reasoning modules to improve performance across diverse games.

Contribution

It introduces a unified modular harness design that allows LLM-based agents to adapt to various gaming environments without domain-specific engineering.

Findings

Performance improves over baseline agents.

Memory is crucial in long-horizon puzzles.

Perception is vital in noisy visual arcade games.

Abstract

We introduce a modular harness design for LLM agents that composes of perception, memory, and reasoning components, enabling a single LLM or VLM backbone to tackle a wide spectrum of multi turn gaming environments without domain-specific engineering. Using classic and modern game suites as low-barrier, high-diversity testbeds, our framework provides a unified workflow for analyzing how each module affects performance across dynamic interactive settings. Extensive experiments demonstrate that the harness lifts gameplay performance consistently over un-harnessed baselines and reveals distinct contribution patterns, for example, memory dominates in long-horizon puzzles while perception is critical in vision noisy arcades. These findings highlight the effectiveness of our modular harness design in advancing general-purpose agent, given the familiarity and ubiquity of games in everyday human experience.