Baba is LLM: Reasoning in a Game with Dynamic Rules

TL;DR

This paper evaluates large language models' ability to reason and solve puzzles in the game Baba is You, focusing on dynamic rule manipulation and the models' understanding of game mechanics and reasoning capabilities.

Contribution

It introduces a novel evaluation of LLMs on a complex reasoning task involving dynamic rules, highlighting the limitations and potential of current models.

Findings

Larger models like GPT-4o perform better in reasoning tasks.

Finetuning improves level analysis but not solution formulation.

All models struggle with understanding dynamic rule changes.

Abstract

Large language models (LLMs) are known to perform well on language tasks, but struggle with reasoning tasks. This paper explores the ability of LLMs to play the 2D puzzle game Baba is You, in which players manipulate rules by rearranging text blocks that define object properties. Given that this rule-manipulation relies on language abilities and reasoning, it is a compelling challenge for LLMs. Six LLMs are evaluated using different prompt types, including (1) simple, (2) rule-extended and (3) action-extended prompts. In addition, two models (Mistral, OLMo) are finetuned using textual and structural data from the game. Results show that while larger models (particularly GPT-4o) perform better in reasoning and puzzle solving, smaller unadapted models struggle to recognize game mechanics or apply rule changes. Finetuning improves the ability to analyze the game levels, but does not significantly improve solution formulation. We conclude that even for state-of-the-art and finetuned LLMs, reasoning about dynamic rule changes is difficult (specifically, understanding the use-mention distinction). The results provide insights into the applicability of LLMs to complex problem-solving tasks and highlight the suitability of games with dynamically changing rules for testing reasoning and reflection by LLMs.