GSIFN: A Graph-Structured and Interlaced-Masked Multimodal Transformer-based Fusion Network for Multimodal Sentiment Analysis

TL;DR

GSIFN introduces a graph-structured, interlaced-masked multimodal Transformer with a self-supervised framework, achieving superior multimodal sentiment analysis performance while reducing computational overhead.

Contribution

The paper proposes GSIFN, a novel multimodal fusion network that effectively balances representation capability and efficiency using graph-structured and interlaced-masked Transformer components.

Findings

Outperforms previous state-of-the-art models on CMU-MOSI, CMU-MOSEI, and CH-SIMS datasets.

Achieves higher accuracy with significantly lower computational overhead.

Demonstrates robustness and efficiency in multimodal sentiment analysis.

Abstract

Multimodal Sentiment Analysis (MSA) leverages multiple data modals to analyze human sentiment. Existing MSA models generally employ cutting-edge multimodal fusion and representation learning-based methods to promote MSA capability. However, there are two key challenges: (i) in existing multimodal fusion methods, the decoupling of modal combinations and tremendous parameter redundancy, lead to insufficient fusion performance and efficiency; (ii) a challenging trade-off exists between representation capability and computational overhead in unimodal feature extractors and encoders. Our proposed GSIFN incorporates two main components to solve these problems: (i) a graph-structured and interlaced-masked multimodal Transformer. It adopts the Interlaced Mask mechanism to construct robust multimodal graph embedding, achieve all-modal-in-one Transformer-based fusion, and greatly reduce the computational overhead; (ii) a self-supervised learning framework with low computational overhead and high performance, which utilizes a parallelized LSTM with matrix memory to enhance non-verbal modal features for unimodal label generation. Evaluated on the MSA datasets CMU-MOSI, CMU-MOSEI, and CH-SIMS, GSIFN demonstrates superior performance with significantly lower computational overhead compared with previous state-of-the-art models.