Transformer-Driven Multimodal Fusion for Explainable Suspiciousness Estimation in Visual Surveillance

TL;DR

This paper introduces a large-scale dataset and a transformer-based multimodal framework for real-time suspiciousness estimation in visual surveillance, enhancing accuracy and interpretability in threat detection.

Contribution

It presents the USE50k dataset and DeepUSEvision framework, combining multimodal fusion and transformer networks for improved suspiciousness analysis in complex environments.

Findings

Superior accuracy over state-of-the-art methods

Robustness across diverse surveillance scenarios

Enhanced interpretability of suspiciousness scores

Abstract

Suspiciousness estimation is critical for proactive threat detection and ensuring public safety in complex environments. This work introduces a large-scale annotated dataset, USE50k, along with a computationally efficient vision-based framework for real-time suspiciousness analysis. The USE50k dataset contains 65,500 images captured from diverse and uncontrolled environments, such as airports, railway stations, restaurants, parks, and other public areas, covering a broad spectrum of cues including weapons, fire, crowd density, abnormal facial expressions, and unusual body postures. Building on this dataset, we present DeepUSEvision, a lightweight and modular system integrating three key components, i.e., a Suspicious Object Detector based on an enhanced YOLOv12 architecture, dual Deep Convolutional Neural Networks (DCNN-I and DCNN-II) for facial expression and body-language recognition using image and landmark features, and a transformer-based Discriminator Network that adaptively fuses multimodal outputs to yield an interpretable suspiciousness score. Extensive experiments confirm the superior accuracy, robustness, and interpretability of the proposed framework compared to state-of-the-art approaches. Collectively, the USE50k dataset and the DeepUSEvision framework establish a strong and scalable foundation for intelligent surveillance and real-time risk assessment in safety-critical applications.