PCGRLLM: Large Language Model-Driven Reward Design for Procedural Content Generation Reinforcement Learning

TL;DR

This paper introduces PCGRLLM, a novel approach using large language models with feedback and reasoning techniques to automate reward design in game AI, significantly reducing human effort and improving performance.

Contribution

The work presents an extended architecture for reward generation employing LLMs with feedback and reasoning prompts, demonstrating improved performance in content generation tasks.

Findings

Achieved 415% performance improvement with one LLM

Achieved 40% performance improvement with another LLM

Demonstrated generalizability across environments and models

Abstract

Reward design plays a pivotal role in the training of game AIs, requiring substantial domain-specific knowledge and human effort. In recent years, several studies have explored reward generation for training game agents and controlling robots using large language models (LLMs). In the content generation literature, there has been early work on generating reward functions for reinforcement learning agent generators. This work introduces PCGRLLM, an extended architecture based on earlier work, which employs a feedback mechanism and several reasoning-based prompt engineering techniques. We evaluate the proposed method on a story-to-reward generation task in a two-dimensional environment using two state-of-the-art LLMs, demonstrating the generalizability of our approach. Our experiments provide insightful evaluations that demonstrate the capabilities of LLMs essential for content generation tasks. The results highlight significant performance improvements of 415% and 40% respectively, depending on the zero-shot capabilities of the language model. Our work demonstrates the potential to reduce human dependency in game AI development, while supporting and enhancing creative processes.