Structures Meet Semantics: Multimodal Fusion via Graph Contrastive Learning

TL;DR

This paper introduces the Structural-Semantic Unifier (SSU), a novel multimodal fusion framework that leverages modality-specific graphs and contrastive learning to improve sentiment analysis by capturing structural dependencies and semantic alignment.

Contribution

The paper proposes SSU, a new framework that integrates structural and semantic information across modalities using graph construction and contrastive learning, advancing multimodal sentiment analysis.

Findings

SSU achieves state-of-the-art results on CMU-MOSI and CMU-MOSEI datasets.

SSU reduces computational overhead compared to previous methods.

Qualitative analysis shows improved interpretability and nuanced emotional pattern capture.

Abstract

Multimodal sentiment analysis (MSA) aims to infer emotional states by effectively integrating textual, acoustic, and visual modalities. Despite notable progress, existing multimodal fusion methods often neglect modality-specific structural dependencies and semantic misalignment, limiting their quality, interpretability, and robustness. To address these challenges, we propose a novel framework called the Structural-Semantic Unifier (SSU), which systematically integrates modality-specific structural information and cross-modal semantic grounding for enhanced multimodal representations. Specifically, SSU dynamically constructs modality-specific graphs by leveraging linguistic syntax for text and a lightweight, text-guided attention mechanism for acoustic and visual modalities, thus capturing detailed intra-modal relationships and semantic interactions. We further introduce a semantic anchor, derived from global textual semantics, that serves as a cross-modal alignment hub, effectively harmonizing heterogeneous semantic spaces across modalities. Additionally, we develop a multiview contrastive learning objective that promotes discriminability, semantic consistency, and structural coherence across intra- and inter-modal views. Extensive evaluations on two widely used benchmark datasets, CMU-MOSI and CMU-MOSEI, demonstrate that SSU consistently achieves state-of-the-art performance while significantly reducing computational overhead compared to prior methods. Comprehensive qualitative analyses further validate SSU's interpretability and its ability to capture nuanced emotional patterns through semantically grounded interactions.