A Video Anomaly Detection Framework based on Appearance-Motion Semantics Representation Consistency

TL;DR

This paper introduces AMSRC, a novel video anomaly detection framework that leverages the consistency between appearance and motion semantics to identify anomalies more effectively, especially in surveillance videos.

Contribution

The paper proposes a two-stream encoder with constraints to enhance appearance-motion semantic consistency for improved anomaly detection performance.

Findings

Effective detection of anomalies with low appearance-motion consistency

Higher reconstruction errors for anomalous samples

Outperforms existing methods in experimental evaluations

Abstract

Video anomaly detection refers to the identification of events that deviate from the expected behavior. Due to the lack of anomalous samples in training, video anomaly detection becomes a very challenging task. Existing methods almost follow a reconstruction or future frame prediction mode. However, these methods ignore the consistency between appearance and motion information of samples, which limits their anomaly detection performance. Anomalies only occur in the moving foreground of surveillance videos, so the semantics expressed by video frame sequences and optical flow without background information in anomaly detection should be highly consistent and significant for anomaly detection. Based on this idea, we propose Appearance-Motion Semantics Representation Consistency (AMSRC), a framework that uses normal data's appearance and motion semantic representation consistency to handle anomaly detection. Firstly, we design a two-stream encoder to encode the appearance and motion information representations of normal samples and introduce constraints to further enhance the consistency of the feature semantics between appearance and motion information of normal samples so that abnormal samples with low consistency appearance and motion feature representation can be identified. Moreover, the lower consistency of appearance and motion features of anomalous samples can be used to generate predicted frames with larger reconstruction error, which makes anomalies easier to spot. Experimental results demonstrate the effectiveness of the proposed method.