SAGE: Semantic-Aware Gray-Box Game Regression Testing with Large Language Models

TL;DR

SAGE introduces a semantic-aware gray-box regression testing framework for games that leverages large language models to automate test generation, optimize test suites, and prioritize relevant tests, reducing costs and improving bug detection.

Contribution

The paper presents SAGE, a novel framework using LLM-guided reinforcement learning and semantic analysis for efficient, automated regression testing in gray-box game environments, addressing key manual and redundancy challenges.

Findings

SAGE outperforms baseline methods in bug detection across tested environments.

SAGE reduces testing costs significantly compared to traditional approaches.

SAGE adapts effectively to version updates, maintaining high testing relevance.

Abstract

The rapid iteration cycles of modern live-service games make regression testing indispensable for maintaining quality and stability. However, existing regression testing approaches face critical limitations, especially in common gray-box settings where full source code access is unavailable: they heavily rely on manual effort for test case construction, struggle to maintain growing suites plagued by redundancy, and lack efficient mechanisms for prioritizing relevant tests. These challenges result in excessive testing costs, limited automation, and insufficient bug detection. To address these issues, we propose SAGE, a semanticaware regression testing framework for gray-box game environments. SAGE systematically addresses the core challenges of test generation, maintenance, and selection. It employs LLM-guided reinforcement learning for efficient, goal-oriented exploration to automatically generate a diverse foundational test suite. Subsequently, it applies a semantic-based multi-objective optimization to refine this suite into a compact, high-value subset by balancing cost, coverage, and rarity. Finally, it leverages LLM-based semantic analysis of update logs to prioritize test cases most relevant to version changes, enabling efficient adaptation across iterations. We evaluate SAGE on two representative environments, Overcooked Plus and Minecraft, comparing against both automated baselines and human-recorded test cases. Across all environments, SAGE achieves superior bug detection with significantly lower execution cost, while demonstrating strong adaptability to version updates.