Invariant Representation Guided Multimodal Sentiment Decoding with Sequential Variation Regularization

TL;DR

This paper introduces a novel multimodal sentiment analysis method that enhances temporal and modality stability through invariant fusion and sequential variation regularization, leading to more consistent sentiment predictions across diverse data.

Contribution

It proposes a dual enhancement strategy with modality-invariant fusion and sequential variation regularization to improve stability in multimodal sentiment analysis.

Findings

Effective in capturing stable cross-modal representations

Improves prediction consistency across diverse modalities

Validated on three public datasets

Abstract

Achieving consistent sentiment representation across diverse modalities remains a key challenge in multimodal sentiment analysis. However, rapid emotional fluctuations over time often introduce instability, leading to compromised prediction performance. To address this challenge, we propose a robust sentiment representation dual enhancement strategy that simultaneously enhances the temporal and modality dimensions, guided by targeted mechanisms in both forward and backward propagation. Specifically, in the modality dimension, we introduce a modality invariant fusion mechanism that fosters stable cross-modal representations, which aim to capture the common and stable representations shared across different modalities. In the temporal dimension, we impose a specialized sequential variation regularization term that regulates the model's learning trajectory during backward propagation, which is essentially total variation regularization degenerated into one-dimensional linear differences. Extensive experiments on three standard public datasets validate the effectiveness of our proposed approach.