TempGlitch: Evaluating Vision-Language Models for Temporal Glitch Detection in Gameplay Videos

TL;DR

TempGlitch introduces a benchmark for evaluating vision-language models' ability to detect temporal glitches in gameplay videos, highlighting current models' limitations in temporal reasoning.

Contribution

The paper presents TempGlitch, a novel benchmark for systematic evaluation of VLMs on temporal glitch detection in videos, revealing their current shortcomings.

Findings

Current VLMs perform near chance on TempGlitch.

Denser frame sampling and larger models do not significantly improve detection.

Temporal glitches are harder to detect than spatial ones for VLMs.

Abstract

Vision-language models (VLMs) are increasingly being explored for video game quality assurance, especially gameplay glitch detection. Most existing evaluations, however, treat glitches as static visual anomalies, asking models to detect failures from a single frame. We argue that this framing misses a key distinction: some glitches are spatial and visible in an isolated frame, whereas others are temporal and become evident only through changes across ordered frames. A preliminary study confirms this gap, showing that temporal glitches are substantially harder for VLMs to detect than spatial ones. To enable systematic evaluation of this underexplored setting, we introduce TempGlitch, a controlled gameplay video benchmark for temporal glitch detection. TempGlitch covers five temporal glitch types with balanced per-category samples, together with paired glitch-free videos that enable reliable binary evaluation. We evaluate 12 proprietary and open-weight VLMs across multiple frame-sampling settings. Our results show that current VLMs remain near chance on TempGlitch, often collapsing into either overly conservative behavior that misses most glitches or overly sensitive behavior that flags clean videos as glitchy. Moreover, denser frame sampling and larger model size do not reliably resolve these failures. TempGlitch provides a focused testbed for temporal reasoning, robust gameplay understanding, and automated glitch detection with VLMs. Code and data are available at the project website.