Leveraging Large Language Models for Efficient Failure Analysis in Game Development

TL;DR

This paper introduces a novel method using Large Language Models to automatically identify code changes responsible for test failures in game development, significantly reducing debugging time and improving bug detection accuracy.

Contribution

The paper presents a new LLM-based approach for pinpointing failure-inducing code changes, demonstrating its effectiveness in a large-scale game development environment.

Findings

Achieved 71% accuracy in identifying failure causes.

Reduced debugging time by up to 60%.

Validated effectiveness through quantitative and user studies.

Abstract

In games, and more generally in the field of software development, early detection of bugs is vital to maintain a high quality of the final product. Automated tests are a powerful tool that can catch a problem earlier in development by executing periodically. As an example, when new code is submitted to the code base, a new automated test verifies these changes. However, identifying the specific change responsible for a test failure becomes harder when dealing with batches of changes -- especially in the case of a large-scale project such as a AAA game, where thousands of people contribute to a single code base. This paper proposes a new approach to automatically identify which change in the code caused a test to fail. The method leverages Large Language Models (LLMs) to associate error messages with the corresponding code changes causing the failure. We investigate the effectiveness of our approach with quantitative and qualitative evaluations. Our approach reaches an accuracy of 71% in our newly created dataset, which comprises issues reported by developers at EA over a period of one year. We further evaluated our model through a user study to assess the utility and usability of the tool from a developer perspective, resulting in a significant reduction in time -- up to 60% -- spent investigating issues.