From Code to Play: Benchmarking Program Search for Games Using Large Language Models

TL;DR

This paper evaluates large language models' ability to generate usable code for various game-related tasks in Python and Java, using an evolutionary hill-climbing approach, and finds that model performance varies more by task than size.

Contribution

It introduces a framework combining LLMs with evolutionary algorithms to synthesize game code across multiple tasks and languages, highlighting the importance of model diversity over size.

Findings

Model performance varies more by task than by size.

Larger models produce more executable code but not necessarily higher quality.

Using multiple models and selecting the best results improves reliability.

Abstract

Large language models (LLMs) have shown impressive capabilities in generating program code, opening exciting opportunities for applying program synthesis to games. In this work, we explore the potential of LLMs to directly synthesize usable code for a wide range of gaming applications, focusing on two programming languages, Python and Java. We use an evolutionary hill-climbing algorithm, where the mutations and seeds of the initial programs are controlled by LLMs. For Python, the framework covers various game-related tasks, including five miniature versions of Atari games, ten levels of Baba is You, an environment inspired by Asteroids, and a maze generation task. For Java, the framework contains 12 games from the TAG tabletop games framework. Across 29 tasks, we evaluated 12 language models for Python and 8 for Java. Our findings suggest that the performance of LLMs depends more on the task than on model size. While larger models generate more executable programs, these do not always result in higher-quality solutions but are much more expensive. No model has a clear advantage, although on any specific task, one model may be better. Trying many models on a problem and using the best results across them is more reliable than using just one.