ICYM2I: The illusion of multimodal informativeness under missingness

TL;DR

This paper investigates how missing data in multimodal learning environments can bias information gain estimates and introduces a correction framework, ICYM2I, to address this issue across various datasets.

Contribution

The paper formalizes the problem of missingness in multimodal data and proposes ICYM2I, a novel inverse probability weighting framework to correct bias in information gain estimation.

Findings

Bias occurs when missingness is not accounted for in multimodal data.

ICYM2I effectively corrects for missingness bias in synthetic and real datasets.

Proper adjustment significantly improves the estimation of information gain.

Abstract

Multimodal learning is of continued interest in artificial intelligence-based applications, motivated by the potential information gain from combining different data modalities. However, modalities observed in the source environment may differ from the modalities observed in the target environment due to multiple factors, including cost, hardware failure, or the perceived \textit{informativeness} of a given modality. This change in missingness patterns between the source and target environment has not been carefully studied. Na{\"i}ve estimation of the information gain associated with including an additional modality without accounting for missingness may result in improper estimates of that modality's value in the target environment. We formalize the problem of missingness, demonstrate its ubiquity, and show that the subsequent distribution shift induces bias when the missingness process is not explicitly accounted for. To address this issue, we introduce ICYM2I (In Case You Multimodal Missed It), a framework for the evaluation of predictive performance and information gain under missingness through inverse probability weighting-based correction. We demonstrate the importance of the proposed adjustment to estimate information gain under missingness on synthetic, semi-synthetic, and real-world datasets.