Improving Multimodal Sentiment Analysis via Modality Optimization and Dynamic Primary Modality Selection

TL;DR

This paper introduces MODS, a framework for multimodal sentiment analysis that dynamically selects primary modalities and reduces noise, leading to improved performance across benchmark datasets.

Contribution

The paper proposes a novel framework with a graph-based compressor, adaptive modality selector, and cross-attention module for better multimodal sentiment analysis.

Findings

MODS outperforms existing methods on four benchmark datasets.

Dynamic modality selection improves sentiment prediction accuracy.

Redundancy reduction enhances the quality of multimodal representations.

Abstract

Multimodal Sentiment Analysis (MSA) aims to predict sentiment from language, acoustic, and visual data in videos. However, imbalanced unimodal performance often leads to suboptimal fused representations. Existing approaches typically adopt fixed primary modality strategies to maximize dominant modality advantages, yet fail to adapt to dynamic variations in modality importance across different samples. Moreover, non-language modalities suffer from sequential redundancy and noise, degrading model performance when they serve as primary inputs. To address these issues, this paper proposes a modality optimization and dynamic primary modality selection framework (MODS). First, a Graph-based Dynamic Sequence Compressor (GDC) is constructed, which employs capsule networks and graph convolution to reduce sequential redundancy in acoustic/visual modalities. Then, we develop a sample-adaptive Primary Modality Selector (MSelector) for dynamic dominance determination. Finally, a Primary-modality-Centric Cross-Attention (PCCA) module is designed to enhance dominant modalities while facilitating cross-modal interaction. Extensive experiments on four benchmark datasets demonstrate that MODS outperforms state-of-the-art methods, achieving superior performance by effectively balancing modality contributions and eliminating redundant noise.