MIDG: Mixture of Invariant Experts with knowledge injection for Domain Generalization in Multimodal Sentiment Analysis

TL;DR

This paper introduces MIDG, a novel multimodal sentiment analysis framework that enhances domain generalization by extracting invariant features and injecting cross-modal knowledge, leading to improved performance across datasets.

Contribution

The paper proposes a Mixture of Invariant Experts model combined with a Cross-Modal Adapter to better capture inter-modal synergies and semantic richness in domain-generalized MSA.

Findings

MIDG outperforms existing methods on three datasets.

Enhanced inter-modal feature extraction improves sentiment analysis accuracy.

Cross-modal knowledge injection enriches multimodal representations.

Abstract

Existing methods in domain generalization for Multimodal Sentiment Analysis (MSA) often overlook inter-modal synergies during invariant features extraction, which prevents the accurate capture of the rich semantic information within multimodal data. Additionally, while knowledge injection techniques have been explored in MSA, they often suffer from fragmented cross-modal knowledge, overlooking specific representations that exist beyond the confines of unimodal. To address these limitations, we propose a novel MSA framework designed for domain generalization. Firstly, the framework incorporates a Mixture of Invariant Experts model to extract domain-invariant features, thereby enhancing the model's capacity to learn synergistic relationships between modalities. Secondly, we design a Cross-Modal Adapter to augment the semantic richness of multimodal representations through cross-modal knowledge injection. Extensive domain experiments conducted on three datasets demonstrate that the proposed MIDG achieves superior performance.