A cross-modal fusion network based on self-attention and residual structure for multimodal emotion recognition

TL;DR

This paper introduces a novel cross-modal fusion network using self-attention and residual structures for multimodal emotion recognition, effectively preserving semantic information and enhancing performance on the RAVDESS dataset.

Contribution

The paper proposes a new fusion network that combines self-attention and residual structures to improve multimodal emotion recognition by maintaining semantic integrity.

Findings

Achieves 75.76% accuracy on RAVDESS dataset

Outperforms existing methods in multimodal emotion recognition

Uses 26.30 million parameters for the model

Abstract

The audio-video based multimodal emotion recognition has attracted a lot of attention due to its robust performance. Most of the existing methods focus on proposing different cross-modal fusion strategies. However, these strategies introduce redundancy in the features of different modalities without fully considering the complementary properties between modal information, and these approaches do not guarantee the non-loss of original semantic information during intra- and inter-modal interactions. In this paper, we propose a novel cross-modal fusion network based on self-attention and residual structure (CFN-SR) for multimodal emotion recognition. Firstly, we perform representation learning for audio and video modalities to obtain the semantic features of the two modalities by efficient ResNeXt and 1D CNN, respectively. Secondly, we feed the features of the two modalities into the cross-modal blocks separately to ensure efficient complementarity and completeness of information through the self-attention mechanism and residual structure. Finally, we obtain the output of emotions by splicing the obtained fused representation with the original representation. To verify the effectiveness of the proposed method, we conduct experiments on the RAVDESS dataset. The experimental results show that the proposed CFN-SR achieves the state-of-the-art and obtains 75.76% accuracy with 26.30M parameters. Our code is available at https://github.com/skeletonNN/CFN-SR.