Complementary Fusion of Multi-Features and Multi-Modalities in Sentiment Analysis

TL;DR

This paper introduces DFF-ATMF, a novel multimodal sentiment analysis model that fuses multi-feature and multi-modality information from audio and text, achieving state-of-the-art results on multiple datasets.

Contribution

The paper proposes a new fusion strategy combining multi-feature and multi-modality attention mechanisms for improved multimodal sentiment analysis.

Findings

DFF-ATMF achieves competitive results on CMU-MOSI and CMU-MOSEI datasets.

The deep features learned are shown to be complementary and robust.

The model sets new state-of-the-art on the IEMOCAP dataset.

Abstract

Sentiment analysis, mostly based on text, has been rapidly developing in the last decade and has attracted widespread attention in both academia and industry. However, the information in the real world usually comes from multiple modalities, such as audio and text. Therefore, in this paper, based on audio and text, we consider the task of multimodal sentiment analysis and propose a novel fusion strategy including both multi-feature fusion and multi-modality fusion to improve the accuracy of audio-text sentiment analysis. We call it the DFF-ATMF (Deep Feature Fusion - Audio and Text Modality Fusion) model, which consists of two parallel branches, the audio modality based branch and the text modality based branch. Its core mechanisms are the fusion of multiple feature vectors and multiple modality attention. Experiments on the CMU-MOSI dataset and the recently released CMU-MOSEI dataset, both collected from YouTube for sentiment analysis, show the very competitive results of our DFF-ATMF model. Furthermore, by virtue of attention weight distribution heatmaps, we also demonstrate the deep features learned by using DFF-ATMF are complementary to each other and robust. Surprisingly, DFF-ATMF also achieves new state-of-the-art results on the IEMOCAP dataset, indicating that the proposed fusion strategy also has a good generalization ability for multimodal emotion recognition.