Baba Is AI: Break the Rules to Beat the Benchmark

TL;DR

This paper introduces a new benchmark based on the game Baba Is You to evaluate AI's ability to creatively manipulate rules, revealing significant limitations of current large language models in generalizing rule-based problem solving.

Contribution

The paper presents a novel benchmark for testing AI creativity and rule manipulation, highlighting the gap in current models' ability to generalize in rule-changing scenarios.

Findings

State-of-the-art models fail to generalize when manipulating game rules.

Models perform well on standard tasks but struggle with rule redefinition.

Current models lack the creative flexibility needed for rule-based problem solving.

Abstract

Humans solve problems by following existing rules and procedures, and also by leaps of creativity to redefine those rules and objectives. To probe these abilities, we developed a new benchmark based on the game Baba Is You where an agent manipulates both objects in the environment and rules, represented by movable tiles with words written on them, to reach a specified goal and win the game. We test three state-of-the-art multi-modal large language models (OpenAI GPT-4o, Google Gemini-1.5-Pro and Gemini-1.5-Flash) and find that they fail dramatically when generalization requires that the rules of the game must be manipulated and combined.