Multi-scale Bottleneck Transformer for Weakly Supervised Multimodal Violence Detection

TL;DR

This paper introduces a multi-scale bottleneck transformer approach for weakly supervised multimodal violence detection, effectively addressing challenges like redundancy, imbalance, and asynchrony across modalities, and achieves state-of-the-art results.

Contribution

It proposes a novel MSBT fusion module with bottleneck tokens and a temporal consistency contrast loss for improved multimodal violence detection.

Findings

Achieves state-of-the-art performance on XD-Violence dataset.

Effectively handles modality imbalance and asynchrony.

Outperforms existing methods in weakly supervised settings.

Abstract

Weakly supervised multimodal violence detection aims to learn a violence detection model by leveraging multiple modalities such as RGB, optical flow, and audio, while only video-level annotations are available. In the pursuit of effective multimodal violence detection (MVD), information redundancy, modality imbalance, and modality asynchrony are identified as three key challenges. In this work, we propose a new weakly supervised MVD method that explicitly addresses these challenges. Specifically, we introduce a multi-scale bottleneck transformer (MSBT) based fusion module that employs a reduced number of bottleneck tokens to gradually condense information and fuse each pair of modalities and utilizes a bottleneck token-based weighting scheme to highlight more important fused features. Furthermore, we propose a temporal consistency contrast loss to semantically align pairwise fused features. Experiments on the largest-scale XD-Violence dataset demonstrate that the proposed method achieves state-of-the-art performance. Code is available at https://github.com/shengyangsun/MSBT.