Just Dance with $\pi$! A Poly-modal Inductor for Weakly-supervised Video Anomaly Detection

TL;DR

This paper introduces PI-VAD, a multi-modal framework that enhances video anomaly detection by integrating five additional modalities with RGB features, achieving state-of-the-art results in real-world scenarios.

Contribution

The paper proposes a novel multi-modal VAD framework with plug-in modules that induce multi-modal cues into RGB features, improving robustness without extra inference overhead.

Findings

Achieves state-of-the-art accuracy on three VAD datasets.

Effectively integrates five modalities during training only.

Improves detection reliability in complex real-world scenarios.

Abstract

Weakly-supervised methods for video anomaly detection (VAD) are conventionally based merely on RGB spatio-temporal features, which continues to limit their reliability in real-world scenarios. This is due to the fact that RGB-features are not sufficiently distinctive in setting apart categories such as shoplifting from visually similar events. Therefore, towards robust complex real-world VAD, it is essential to augment RGB spatio-temporal features by additional modalities. Motivated by this, we introduce the Poly-modal Induced framework for VAD: "PI-VAD", a novel approach that augments RGB representations by five additional modalities. Specifically, the modalities include sensitivity to fine-grained motion (Pose), three dimensional scene and entity representation (Depth), surrounding objects (Panoptic masks), global motion (optical flow), as well as language cues (VLM). Each modality represents an axis of a polygon, streamlined to add salient cues to RGB. PI-VAD includes two plug-in modules, namely Pseudo-modality Generation module and Cross Modal Induction module, which generate modality-specific prototypical representation and, thereby, induce multi-modal information into RGB cues. These modules operate by performing anomaly-aware auxiliary tasks and necessitate five modality backbones -- only during training. Notably, PI-VAD achieves state-of-the-art accuracy on three prominent VAD datasets encompassing real-world scenarios, without requiring the computational overhead of five modality backbones at inference.