Learning Invariant Modality Representation for Robust Multimodal Learning from a Causal Inference Perspective

TL;DR

This paper introduces a causal invariant representation learning framework for multimodal affective computing, improving robustness and generalization across environments and noisy data.

Contribution

It proposes a theoretically grounded disentanglement method to separate causal invariant and spurious representations in multimodal data.

Findings

Achieves state-of-the-art performance on multiple benchmarks.

Excels in out-of-distribution and noisy data scenarios.

Ensures stable predictive relationships across environments.

Abstract

Multimodal affective computing aims to predict humans' sentiment, emotion, intention, and opinion using language, acoustic, and visual modalities. However, current models often learn spurious correlations that harm generalization under distribution shifts or noisy modalities. To address this, we propose a causal modality-invariant representation (CmIR) learning framework for robust multimodal learning. At its core, we introduce a theoretically grounded disentanglement method that separates each modality into `causal invariant representation' and `environment-specific spurious representation' from a causal inference perspective. CmIR ensures that the learned invariant representations retain stable predictive relationships with labels across different environments while preserving sufficient information from the raw inputs via invariance constraint, mutual information constraint, and reconstruction constraint. Experiments across multiple multimodal benchmarks demonstrate that CmIR achieves state-of-the-art performance. CmIR particularly excels on out-of-distribution data and noisy data, confirming its robustness and generalizability.