Weak Supervision for Label Efficient Visual Bug Detection

TL;DR

This paper introduces a weak supervision approach using unlabeled gameplay data and domain-specific augmentations to improve visual bug detection in video games, especially in low-data scenarios.

Contribution

It presents a novel weak supervision method leveraging unlabeled gameplay and self-supervised objectives for effective visual bug detection in video games.

Findings

Outperforms supervised baseline in low-data regime (F1 score 0.336 to 0.550)

Achieves high accuracy with only 5 labeled exemplars

Demonstrates adaptability across various visual bugs

Abstract

As video games evolve into expansive, detailed worlds, visual quality becomes essential, yet increasingly challenging. Traditional testing methods, limited by resources, face difficulties in addressing the plethora of potential bugs. Machine learning offers scalable solutions; however, heavy reliance on large labeled datasets remains a constraint. Addressing this challenge, we propose a novel method, utilizing unlabeled gameplay and domain-specific augmentations to generate datasets & self-supervised objectives used during pre-training or multi-task settings for downstream visual bug detection. Our methodology uses weak-supervision to scale datasets for the crafted objectives and facilitates both autonomous and interactive weak-supervision, incorporating unsupervised clustering and/or an interactive approach based on text and geometric prompts. We demonstrate on first-person player clipping/collision bugs (FPPC) within the expansive Giantmap game world, that our approach is very effective, improving over a strong supervised baseline in a practical, very low-prevalence, low data regime (0.336 $\rightarrow$ 0.550 F1 score). With just 5 labeled "good" exemplars (i.e., 0 bugs), our self-supervised objective alone captures enough signal to outperform the low-labeled supervised settings. Building on large-pretrained vision models, our approach is adaptable across various visual bugs. Our results suggest applicability in curating datasets for broader image and video tasks within video games beyond visual bugs.