Enhancing Human Action Recognition and Violence Detection Through Deep Learning Audiovisual Fusion

TL;DR

This paper introduces a deep learning audiovisual fusion approach for improved human activity recognition and violence detection, demonstrating high accuracy and real-world applicability in public safety scenarios.

Contribution

It presents a hybrid fusion-based deep learning model combining audio and video modalities, outperforming existing methods on a new violence detection dataset.

Findings

96.67% accuracy on validation data

Successfully detected 52 out of 54 real-world videos

Effective in real-world violence detection scenarios

Abstract

This paper proposes a hybrid fusion-based deep learning approach based on two different modalities, audio and video, to improve human activity recognition and violence detection in public places. To take advantage of audiovisual fusion, late fusion, intermediate fusion, and hybrid fusion-based deep learning (HFBDL) are used and compared. Since the objective is to detect and recognize human violence in public places, Real-life violence situation (RLVS) dataset is expanded and used. Simulating results of HFBDL show 96.67\% accuracy on validation data, which is more accurate than the other state-of-the-art methods on this dataset. To showcase our model's ability in real-world scenarios, another dataset of 54 sounded videos of both violent and non-violent situations was recorded. The model could successfully detect 52 out of 54 videos correctly. The proposed method shows a promising performance on real scenarios. Thus, it can be used for human action recognition and violence detection in public places for security purposes.