MSAF: Multimodal Split Attention Fusion

TL;DR

This paper introduces MSAF, a novel multimodal fusion module that emphasizes important features across modalities, compatible with various neural network architectures, and demonstrates improved performance in emotion, sentiment, and action recognition tasks.

Contribution

The paper presents MSAF, a flexible and effective multimodal fusion module that can be integrated into different neural networks and leverages pretrained models for enhanced multimodal learning.

Findings

MSAF improves accuracy in emotion recognition.

MSAF outperforms existing multimodal fusion methods.

The module is compatible with CNNs and RNNs.

Abstract

Multimodal learning mimics the reasoning process of the human multi-sensory system, which is used to perceive the surrounding world. While making a prediction, the human brain tends to relate crucial cues from multiple sources of information. In this work, we propose a novel multimodal fusion module that learns to emphasize more contributive features across all modalities. Specifically, the proposed Multimodal Split Attention Fusion (MSAF) module splits each modality into channel-wise equal feature blocks and creates a joint representation that is used to generate soft attention for each channel across the feature blocks. Further, the MSAF module is designed to be compatible with features of various spatial dimensions and sequence lengths, suitable for both CNNs and RNNs. Thus, MSAF can be easily added to fuse features of any unimodal networks and utilize existing pretrained unimodal model weights. To demonstrate the effectiveness of our fusion module, we design three multimodal networks with MSAF for emotion recognition, sentiment analysis, and action recognition tasks. Our approach achieves competitive results in each task and outperforms other application-specific networks and multimodal fusion benchmarks.