Multi-view and Multi-modal Event Detection Utilizing Transformer-based Multi-sensor fusion

TL;DR

This paper introduces MultiTrans, a Transformer-based multi-sensor fusion method that effectively combines multi-view and multi-modal data for improved event detection in complex real-world environments.

Contribution

The paper presents a novel Transformer-based fusion approach for integrating multi-view and multi-modal sensor data for event detection.

Findings

Improved event detection performance over baseline methods

Effective fusion of multi-view and multi-modal data

Validated on a newly collected dataset

Abstract

We tackle a challenging task: multi-view and multi-modal event detection that detects events in a wide-range real environment by utilizing data from distributed cameras and microphones and their weak labels. In this task, distributed sensors are utilized complementarily to capture events that are difficult to capture with a single sensor, such as a series of actions of people moving in an intricate room, or communication between people located far apart in a room. For sensors to cooperate effectively in such a situation, the system should be able to exchange information among sensors and combines information that is useful for identifying events in a complementary manner. For such a mechanism, we propose a Transformer-based multi-sensor fusion (MultiTrans) which combines multi-sensor data on the basis of the relationships between features of different viewpoints and modalities. In the experiments using a dataset newly collected for this task, our proposed method using MultiTrans improved the event detection performance and outperformed comparatives.