CoLoRSMamba: Conditional LoRA-Steered Mamba for Supervised Multimodal Violence Detection

TL;DR

CoLoRSMamba is a multimodal architecture that enhances violence detection by adaptively combining audio and video data through scene-aware modulation, achieving high accuracy with efficient computation.

Contribution

Introduces CoLoRSMamba, a novel conditional LoRA-based multimodal model that improves violence detection by scene-aware audio-visual integration without token-level cross-attention.

Findings

Outperforms audio-only, video-only, and multimodal baselines on NTU-CCTV and DVD datasets.

Achieves 88.63% accuracy and 86.24% F1-V on NTU-CCTV.

Offers a favorable accuracy-efficiency tradeoff with fewer parameters and FLOPs.

Abstract

Violence detection benefits from audio, but real-world soundscapes can be noisy or weakly related to the visible scene. We present CoLoRSMamba, a directional Video to Audio multimodal architecture that couples VideoMamba and AudioMamba through CLS-guided conditional LoRA. At each layer, the VideoMamba CLS token produces a channel-wise modulation vector and a stabilization gate that adapt the AudioMamba projections responsible for the selective state-space parameters (Delta, B, C), including the step-size pathway, yielding scene-aware audio dynamics without token-level cross-attention. Training combines binary classification with a symmetric AV-InfoNCE objective that aligns clip-level audio and video embeddings. To support fair multimodal evaluation, we curate audio-filtered clip level subsets of the NTU-CCTV and DVD datasets from temporal annotations, retaining only clips with available audio. On these subsets, CoLoRSMamba outperforms representative audio-only, video-only, and multimodal baselines, achieving 88.63% accuracy / 86.24% F1-V on NTU-CCTV and 75.77% accuracy / 72.94% F1-V on DVD. It further offers a favorable accuracy-efficiency tradeoff, surpassing several larger models with fewer parameters and FLOPs.