Automatic Bug Detection in LLM-Powered Text-Based Games Using LLMs

TL;DR

This paper introduces an automated LLM-based method to detect bugs in LLM-powered text-based games by analyzing player logs, addressing a gap in bug detection techniques for such interactive systems.

Contribution

The paper presents a novel systematic approach for automatically identifying bugs in LLM-driven games from logs, improving over unstructured methods and reducing reliance on additional data.

Findings

Effectively identifies bugs in LLM-powered games

Outperforms unstructured bug detection methods

Works without extra data collection

Abstract

Advancements in large language models (LLMs) are revolutionizing interactive game design, enabling dynamic plotlines and interactions between players and non-player characters (NPCs). However, LLMs may exhibit flaws such as hallucinations, forgetfulness, or misinterpretations of prompts, causing logical inconsistencies and unexpected deviations from intended designs. Automated techniques for detecting such game bugs are still lacking. To address this, we propose a systematic LLM-based method for automatically identifying such bugs from player game logs, eliminating the need for collecting additional data such as post-play surveys. Applied to a text-based game DejaBoom!, our approach effectively identifies bugs inherent in LLM-powered interactive games, surpassing unstructured LLM-powered bug-catching methods and filling the gap in automated detection of logical and design flaws.