Audio-visual Representation Learning for Anomaly Events Detection in Crowds

TL;DR

This paper proposes a multi-modal audio-visual learning approach using a two-branch neural network to improve anomaly event detection in crowds, demonstrating that incorporating audio signals enhances detection accuracy over visual-only methods.

Contribution

It introduces a novel two-branch network combining visual and audio features for crowd anomaly detection, outperforming existing state-of-the-art methods on the SHADE dataset.

Findings

Audio signals significantly improve detection accuracy.

The proposed method outperforms existing approaches.

Fusion of audio and visual features enhances robustness.

Abstract

In recent years, anomaly events detection in crowd scenes attracts many researchers' attention, because of its importance to public safety. Existing methods usually exploit visual information to analyze whether any abnormal events have occurred due to only visual sensors are generally equipped in public places. However, when an abnormal event in crowds occurs, sound information may be discriminative to assist the crowd analysis system to determine whether there is an abnormality. Compare with vision information that is easily occluded, audio signals have a certain degree of penetration. Thus, this paper attempt to exploit multi-modal learning for modeling the audio and visual signals simultaneously. To be specific, we design a two-branch network to model different types of information. The first is a typical 3D CNN model to extract temporal appearance features from video clips. The second is an audio CNN for encoding Log Mel-Spectrogram of audio signals. Finally, by fusing the above features, a more accurate prediction will be produced. We conduct the experiments on SHADE dataset, a synthetic audio-visual dataset in surveillance scenes, and find introducing audio signals effectively improves the performance of anomaly events detection and outperforms other state-of-the-art methods. Furthermore, we will release the code and the pre-trained models as soon as possible.