Towards the Causal Complete Cause of Multi-Modal Representation Learning

TL;DR

This paper introduces a causal framework for multi-modal learning representations, emphasizing their sufficiency and necessity, and proposes a regularization method to enforce these properties, improving representation quality.

Contribution

It defines the Causal Complete Cause ($C^3$), explores its identifiability, and develops a regularization method to enforce causal completeness in multi-modal representations.

Findings

The $C^3$ regularization improves representation causality.

The twin network effectively estimates $C^3$ risk.

The method demonstrates superior performance in experiments.

Abstract

Multi-Modal Learning (MML) aims to learn effective representations across modalities for accurate predictions. Existing methods typically focus on modality consistency and specificity to learn effective representations. However, from a causal perspective, they may lead to representations that contain insufficient and unnecessary information. To address this, we propose that effective MML representations should be causally sufficient and necessary. Considering practical issues like spurious correlations and modality conflicts, we relax the exogeneity and monotonicity assumptions prevalent in prior works and explore the concepts specific to MML, i.e., Causal Complete Cause $C^3$. We begin by defining $C^3$, which quantifies the probability of representations being causally sufficient and necessary. We then discuss the identifiability of $C^3$ and introduce an instrumental variable to support identifying $C^3$ with non-exogeneity and non-monotonicity. Building on this, we conduct the $C^3$ measurement, i.e., \(C^3\) risk. We propose a twin network to estimate it through (i) the real-world branch: utilizing the instrumental variable for sufficiency, and (ii) the hypothetical-world branch: applying gradient-based counterfactual modeling for necessity. Theoretical analyses confirm its reliability. Based on these results, we propose $C^3$ Regularization, a plug-and-play method that enforces the causal completeness of the learned representations by minimizing $C^3$ risk. Extensive experiments demonstrate its effectiveness.