PSA-MF: Personality-Sentiment Aligned Multi-Level Fusion for Multimodal Sentiment Analysis

TL;DR

This paper introduces a novel multimodal sentiment analysis framework that incorporates personality traits and multi-level fusion to improve sentiment recognition accuracy across text, visual, and audio data.

Contribution

It proposes a new personality-sentiment alignment method and a multi-level fusion strategy for enhanced multimodal sentiment analysis performance.

Findings

Achieved state-of-the-art results on benchmark datasets.

Effectively incorporates personality traits into sentiment analysis.

Demonstrated improved fusion of multimodal features.

Abstract

Multimodal sentiment analysis (MSA) is a research field that recognizes human sentiments by combining textual, visual, and audio modalities. The main challenge lies in integrating sentiment-related information from different modalities, which typically arises during the unimodal feature extraction phase and the multimodal feature fusion phase. Existing methods extract only shallow information from unimodal features during the extraction phase, neglecting sentimental differences across different personalities. During the fusion phase, they directly merge the feature information from each modality without considering differences at the feature level. This ultimately affects the model's recognition performance. To address this problem, we propose a personality-sentiment aligned multi-level fusion framework. We introduce personality traits during the feature extraction phase and propose a novel personality-sentiment alignment method to obtain personalized sentiment embeddings from the textual modality for the first time. In the fusion phase, we introduce a novel multi-level fusion method. This method gradually integrates sentimental information from textual, visual, and audio modalities through multimodal pre-fusion and a multi-level enhanced fusion strategy. Our method has been evaluated through multiple experiments on two commonly used datasets, achieving state-of-the-art results.