Towards Adaptive Fusion of Multimodal Deep Networks for Human Action Recognition

TL;DR

This paper presents a novel adaptive multimodal fusion approach using gating mechanisms to improve human action recognition accuracy across various datasets and applications.

Contribution

It introduces a new adaptive fusion methodology with gating mechanisms for multimodal deep networks, enhancing recognition performance over traditional unimodal methods.

Findings

Gating-based fusion outperforms unimodal approaches.

Enhanced accuracy in action recognition and violence detection.

Effective across multiple datasets and self-supervised tasks.

Abstract

This study introduces a pioneering methodology for human action recognition by harnessing deep neural network techniques and adaptive fusion strategies across multiple modalities, including RGB, optical flows, audio, and depth information. Employing gating mechanisms for multimodal fusion, we aim to surpass limitations inherent in traditional unimodal recognition methods while exploring novel possibilities for diverse applications. Through an exhaustive investigation of gating mechanisms and adaptive weighting-based fusion architectures, our methodology enables the selective integration of relevant information from various modalities, thereby bolstering both accuracy and robustness in action recognition tasks. We meticulously examine various gated fusion strategies to pinpoint the most effective approach for multimodal action recognition, showcasing its superiority over conventional unimodal methods. Gating mechanisms facilitate the extraction of pivotal features, resulting in a more holistic representation of actions and substantial enhancements in recognition performance. Our evaluations across human action recognition, violence action detection, and multiple self-supervised learning tasks on benchmark datasets demonstrate promising advancements in accuracy. The significance of this research lies in its potential to revolutionize action recognition systems across diverse fields. The fusion of multimodal information promises sophisticated applications in surveillance and human-computer interaction, especially in contexts related to active assisted living.