Predicting Defective Visual Code Changes in a Multi-Language AAA Video Game Project

TL;DR

This paper develops defect prediction models that incorporate visual code metrics for AAA video game projects, demonstrating improved accuracy over traditional textual code models, especially when visual code commits are prevalent.

Contribution

It introduces visual code defect prediction models that include visual metrics and evaluates their effectiveness against textual code models in a multi-language AAA game context.

Findings

Models with visual code features outperform in projects with many visual code commits.

Visual code defect prediction models show higher AUC and MCC scores.

Incorporating visual metrics enhances defect prediction accuracy.

Abstract

Video game development increasingly relies on using visual programming languages as the primary way to build video game features. The aim of using visual programming is to move game logic into the hands of game designers, who may not be as well versed in textual coding. In this paper, we empirically observe that there are more defect-inducing commits containing visual code than textual code in a AAA video game project codebase. This indicates that the existing textual code Just-in-Time (JIT) defect prediction models under evaluation by Electronic Arts (EA) may be ineffective as they do not account for changes in visual code. Thus, we focus our research on constructing visual code defect prediction models that encompass visual code metrics and evaluate the models against defect prediction models that use language agnostic features, and textual code metrics. We test our models using features extracted from the historical codebase of a AAA video game project, as well as the historical codebases of 70 open source projects that use textual and visual code. We find that defect prediction models have better performance overall in terms of the area under the ROC curve (AUC), and Mathews Correlation Coefficient (MCC) when incorporating visual code features for projects that contain more commits with visual code than textual code.