Dual Branch VideoMamba with Gated Class Token Fusion for Violence Detection

TL;DR

This paper introduces Dual Branch VideoMamba with Gated Class Token Fusion, an efficient model combining spatial and temporal features for violence detection, achieving state-of-the-art results on a new comprehensive benchmark.

Contribution

It presents a novel dual-branch architecture with gated fusion and a new benchmark dataset for violence detection, improving accuracy and efficiency.

Findings

Achieves state-of-the-art performance on the new benchmark

Balances accuracy and computational efficiency effectively

Demonstrates the effectiveness of SSMs for real-time surveillance

Abstract

The rapid proliferation of surveillance cameras has increased the demand for automated violence detection. While CNNs and Transformers have shown success in extracting spatio-temporal features, they struggle with long-term dependencies and computational efficiency. We propose Dual Branch VideoMamba with Gated Class Token Fusion (GCTF), an efficient architecture combining a dual-branch design and a state-space model (SSM) backbone where one branch captures spatial features, while the other focuses on temporal dynamics. The model performs continuous fusion via a gating mechanism between the branches to enhance the model's ability to detect violent activities even in challenging surveillance scenarios. We also present a new benchmark by merging RWF-2000, RLVS, SURV and VioPeru datasets in video violence detection, ensuring strict separation between training and testing sets. Experimental results demonstrate that our model achieves state-of-the-art performance on this benchmark and also on DVD dataset which is another novel dataset on video violence detection, offering an optimal balance between accuracy and computational efficiency, demonstrating the promise of SSMs for scalable, near real-time surveillance violence detection.