GMFVAD: Using Grained Multi-modal Feature to Improve Video Anomaly Detection

TL;DR

This paper introduces GMFVAD, a novel multi-modal feature refinement method that leverages detailed text and visual content to improve video anomaly detection accuracy, reducing redundancy and achieving state-of-the-art results.

Contribution

The paper proposes a grained multi-modal feature extraction approach that enhances visual features with caption-based text information to improve anomaly detection performance.

Findings

Achieves state-of-the-art performance on four datasets.

Reduces redundant information in visual features.

Validates effectiveness through ablation experiments.

Abstract

Video anomaly detection (VAD) is a challenging task that detects anomalous frames in continuous surveillance videos. Most previous work utilizes the spatio-temporal correlation of visual features to distinguish whether there are abnormalities in video snippets. Recently, some works attempt to introduce multi-modal information, like text feature, to enhance the results of video anomaly detection. However, these works merely incorporate text features into video snippets in a coarse manner, overlooking the significant amount of redundant information that may exist within the video snippets. Therefore, we propose to leverage the diversity among multi-modal information to further refine the extracted features, reducing the redundancy in visual features, and we propose Grained Multi-modal Feature for Video Anomaly Detection (GMFVAD). Specifically, we generate more grained multi-modal feature based on the video snippet, which summarizes the main content, and text features based on the captions of original video will be introduced to further enhance the visual features of highlighted portions. Experiments show that the proposed GMFVAD achieves state-of-the-art performance on four mainly datasets. Ablation experiments also validate that the improvement of GMFVAD is due to the reduction of redundant information.