Graph-based Interaction Augmentation Network for Robust Multimodal Sentiment Analysis

TL;DR

This paper introduces a graph-based framework for multimodal sentiment analysis that models complex intra- and inter-modality interactions, improving robustness against modality imperfections by leveraging complementary information.

Contribution

It proposes a novel hypergraph and directed graph approach to exploit intra- and inter-modality dependencies, enhancing the extraction of missing semantics in imperfect multimodal data.

Findings

Effective in handling modality imperfections

Outperforms existing methods on MOSI and MOSEI datasets

Enhances robustness of multimodal sentiment analysis

Abstract

The inevitable modality imperfection in real-world scenarios poses significant challenges for Multimodal Sentiment Analysis (MSA). While existing methods tailor reconstruction or joint representation learning strategies to restore missing semantics, they often overlook complex dependencies within and across modalities. Consequently, they fail to fully leverage available modalities to capture complementary semantics. To this end, this paper proposes a novel graph-based framework to exploit both intra- and inter-modality interactions, enabling imperfect samples to derive missing semantics from complementary parts for robust MSA. Specifically, we first devise a learnable hypergraph to model intra-modality temporal dependencies to exploit contextual information within each modality. Then, a directed graph is employed to explore inter-modality correlations based on attention mechanism, capturing complementary information across different modalities. Finally, the knowledge from perfect samples is integrated to supervise our interaction processes, guiding the model toward learning reliable and robust joint representations. Extensive experiments on MOSI and MOSEI datasets demonstrate the effectiveness of our method.