GVGAI-LLM: Evaluating Large Language Model Agents with Infinite Games

TL;DR

GVGAI-LLM is a new benchmark using diverse arcade-style games to evaluate large language models' reasoning, problem-solving, and spatial understanding, revealing current limitations and guiding future improvements.

Contribution

It introduces a scalable, game-based benchmark for assessing LLMs' reasoning and spatial skills, with interpretable metrics and the ability to generate infinite test scenarios.

Findings

LLMs show persistent spatial and logical errors.

Interventions like structured prompting lead to partial improvements.

Benchmark reveals significant gaps in LLM reasoning capabilities.

Abstract

We introduce GVGAI-LLM, a video game benchmark for evaluating the reasoning and problem-solving capabilities of large language models (LLMs). Built on the General Video Game AI framework, it features a diverse collection of arcade-style games designed to test a model's ability to handle tasks that differ from most existing LLM benchmarks. The benchmark leverages a video game description language that enables the rapid creation of new games (including rules and levels), helping to prevent overfitting over time. Each game scene is represented by a compact set of ASCII characters, allowing for efficient processing by language models. GVGAI-LLM defines interpretable metrics, including meaningful step ratio, step efficiency, and overall score, to assess model behavior. Through zero-shot evaluations across 118 games with diverse challenges and skill depth, we reveal persistent limitations of LLMs in spatial reasoning and basic planning. Current models consistently exhibit spatial and logical errors, motivating structured prompting and spatial grounding techniques. Although these interventions lead to partial improvements, the benchmark remains very far from being solved. GVGAI-LLM serves as a reproducible testbed for advancing research on language model capabilities, with a particular emphasis on agentic behavior and spatial reasoning. Furthermore, its ability to generate infinite benchmarks, both manually and procedurally, provides a scalable framework for longitudinal evaluation.