Automated Bug Frame Retrieval from Gameplay Videos Using Vision-Language Models

TL;DR

This paper presents an automated pipeline that extracts keyframes from gameplay videos and uses vision-language models to match them with bug descriptions, significantly reducing manual review effort in game bug triage.

Contribution

The authors introduce a novel automated method combining keyframe extraction and vision-language models for efficient bug report verification in gaming.

Findings

Achieves an F1 score of 0.79 and accuracy of 0.89 in matching bug frames.

Captures bug moments in 98.79% of cases with only 1.90% of original frames.

Performs best in Lighting & Shadow, Physics & Collision, and UI & HUD categories.

Abstract

Modern game studios deliver new builds and patches at a rapid pace, generating thousands of bug reports, many of which embed gameplay videos. To verify and triage these bug reports, developers must watch the submitted videos. This manual review is labour-intensive, slow, and hard to scale. In this paper, we introduce an automated pipeline that reduces each video to a single frame that best matches the reported bug description, giving developers instant visual evidence that pinpoints the bug. Our pipeline begins with FFmpeg for keyframe extraction, reducing each video to a median of just 1.90% of its original frames while still capturing bug moments in 98.79 of cases. These keyframes are then evaluated by a vision--language model (GPT-4o), which ranks them based on how well they match the textual bug description and selects the most representative frame. We evaluated this approach using real-world developer-submitted gameplay videos and JIRA bug reports from a popular First-Person Shooter (FPS) game. The pipeline achieves an overall F1 score of 0.79 and Accuracy of 0.89 for the top-1 retrieved frame. Performance is highest for the Lighting & Shadow (F1 = 0.94), Physics & Collision (0.86), and UI & HUD (0.83) bug categories, and lowest for Animation & VFX (0.51). By replacing video viewing with an immediately informative image, our approach dramatically reduces manual effort and speeds up triage and regression checks, offering practical benefits to quality assurance (QA) teams and developers across the game industry.