Multimodal Transformers are Hierarchical Modal-wise Heterogeneous Graphs

TL;DR

This paper introduces GsiT, a graph-structured multimodal transformer that improves efficiency and performance in multimodal sentiment analysis by modeling MulTs as hierarchical heterogeneous graphs and using an interlaced mask mechanism.

Contribution

It formalizes MulTs as hierarchical modal-wise heterogeneous graphs and proposes GsiT with an interlaced mask for efficient, high-performance multimodal fusion.

Findings

GsiT achieves 1/3 parameter reduction compared to traditional MulTs.

GsiT significantly outperforms traditional MulTs on MSA datasets.

The HMHG concept improves model efficiency and effectiveness.

Abstract

Multimodal Sentiment Analysis (MSA) is a rapidly developing field that integrates multimodal information to recognize sentiments, and existing models have made significant progress in this area. The central challenge in MSA is multimodal fusion, which is predominantly addressed by Multimodal Transformers (MulTs). Although act as the paradigm, MulTs suffer from efficiency concerns. In this work, from the perspective of efficiency optimization, we propose and prove that MulTs are hierarchical modal-wise heterogeneous graphs (HMHGs), and we introduce the graph-structured representation pattern of MulTs. Based on this pattern, we propose an Interlaced Mask (IM) mechanism to design the Graph-Structured and Interlaced-Masked Multimodal Transformer (GsiT). It is formally equivalent to MulTs which achieves an efficient weight-sharing mechanism without information disorder through IM, enabling All-Modal-In-One fusion with only 1/3 of the parameters of pure MulTs. A Triton kernel called Decomposition is implemented to ensure avoiding additional computational overhead. Moreover, it achieves significantly higher performance than traditional MulTs. To further validate the effectiveness of GsiT itself and the HMHG concept, we integrate them into multiple state-of-the-art models and demonstrate notable performance improvements and parameter reduction on widely used MSA datasets.