Autoregressive Denoising Score Matching is a Good Video Anomaly Detector

TL;DR

This paper introduces a novel autoregressive denoising score matching approach for video anomaly detection, effectively addressing local mode blindness and incorporating scene, motion, and appearance cues to improve detection accuracy.

Contribution

It proposes a new noise-conditioned score transformer with scene and motion awareness, and an autoregressive denoising mechanism to enhance anomaly detection in videos.

Findings

Achieves state-of-the-art results on three VAD benchmarks.

Effectively detects unseen anomalies by integrating visual information.

Outperforms existing likelihood-based methods in local anomaly detection.

Abstract

Video anomaly detection (VAD) is an important computer vision problem. Thanks to the mode coverage capabilities of generative models, the likelihood-based paradigm is catching growing interest, as it can model normal distribution and detect out-of-distribution anomalies. However, these likelihood-based methods are blind to the anomalies located in local modes near the learned distribution. To handle these ``unseen" anomalies, we dive into three gaps uniquely existing in VAD regarding scene, motion and appearance. Specifically, we first build a noise-conditioned score transformer for denoising score matching. Then, we introduce a scene-dependent and motion-aware score function by embedding the scene condition of input sequences into our model and assigning motion weights based on the difference between key frames of input sequences. Next, to solve the problem of blindness in principle, we integrate unaffected visual information via a novel autoregressive denoising score matching mechanism for inference. Through autoregressively injecting intensifying Gaussian noise into the denoised data and estimating the corresponding score function, we compare the denoised data with the original data to get a difference and aggregate it with the score function for an enhanced appearance perception and accumulate the abnormal context. With all three gaps considered, we can compute a more comprehensive anomaly indicator. Experiments on three popular VAD benchmarks demonstrate the state-of-the-art performance of our method.