Is Video Anomaly Detection Misframed? Evidence from LLM-Based and Multi-Scene Models

TL;DR

This paper critiques current video anomaly detection methods for overgeneralizing across scenes and emphasizes the need for scene-specific, spatially-aware, and explainable models to better capture normal behavior.

Contribution

It highlights the limitations of existing multi-scene, weakly supervised approaches and advocates for a renewed focus on single-scene, spatially-aware anomaly detection models.

Findings

Current models often respond to semantic anomalies rather than deviations from normality.

Multi-scene models suppress spatial localization and introduce semantic bias.

Single-scene, explainable models better capture local normality patterns.

Abstract

Recent video anomaly detection research has expanded rapidly with an emphasis on general models of normality intended to work across many different scenes. While this focus has led to improvements in scalability and multi-scene generalization, it has also shifted the field away from modeling the scene-specific and context-dependent nature of normal behavior. Contemporary approaches frequently rely on video-level weak supervision and opaque pretrained representations from multi-modal large language models (MLLMs), which encourage models to respond to familiar semantic anomaly categories rather than to deviations from the normal patterns of a particular environment. This trend suppresses spatial localization, introduces semantic bias, and reduces anomaly detection to a form of action recognition. In this paper, we examine whether these prevailing formulations align with the core requirements of real-world VAD, which is typically performed within a single scene where normality is determined by local geometry, semantics, and activity patterns. Through targeted visual analyses and empirical evaluations, we demonstrate the practical consequences of these limitations and show that meaningful progress in VAD requires renewed focus on single-scene, spatially-aware, and explainable formulations that capture the nuanced structure of normality within individual environments.