Interventional Imbalanced Multi-Modal Representation Learning via $\beta$-Generalization Front-Door Criterion

TL;DR

This paper introduces a causal inference-based method for multi-modal representation learning that effectively captures the true causal relationships between modalities and labels, overcoming limitations of existing benchmark approaches.

Contribution

It proposes the $eta$-generalization front-door criterion and a novel network to better explore discriminative knowledge in multi-modal data from a causal perspective.

Findings

Outperforms benchmark methods in discriminative tasks

Provides theoretical analysis supporting the causal approach

Demonstrates improved interpretability and exploration capability

Abstract

Multi-modal methods establish comprehensive superiority over uni-modal methods. However, the imbalanced contributions of different modalities to task-dependent predictions constantly degrade the discriminative performance of canonical multi-modal methods. Based on the contribution to task-dependent predictions, modalities can be identified as predominant and auxiliary modalities. Benchmark methods raise a tractable solution: augmenting the auxiliary modality with a minor contribution during training. However, our empirical explorations challenge the fundamental idea behind such behavior, and we further conclude that benchmark approaches suffer from certain defects: insufficient theoretical interpretability and limited exploration capability of discriminative knowledge. To this end, we revisit multi-modal representation learning from a causal perspective and build the Structural Causal Model. Following the empirical explorations, we determine to capture the true causality between the discriminative knowledge of predominant modality and predictive label while considering the auxiliary modality. Thus, we introduce the $\beta$-generalization front-door criterion. Furthermore, we propose a novel network for sufficiently exploring multi-modal discriminative knowledge. Rigorous theoretical analyses and various empirical evaluations are provided to support the effectiveness of the innate mechanism behind our proposed method.