Uncertainty-Aware Multimodal Learning via Conformal Shapley Intervals

TL;DR

This paper introduces conformal Shapley intervals, a method combining Shapley values and conformal inference to quantify uncertainty in multimodal data importance, enabling reliable modality selection with strong performance guarantees.

Contribution

It presents a novel framework for uncertainty-aware modality importance estimation and a provably optimal modality selection procedure in multimodal learning.

Findings

Effective uncertainty quantification for modalities.

Strong predictive performance with few modalities.

Provable guarantees for modality subset selection.

Abstract

Multimodal learning combines information from multiple data modalities to improve predictive performance. However, modalities often contribute unequally and in a data dependent way, making it unclear which data modalities are genuinely informative and to what extent their contributions can be trusted. Quantifying modality level importance together with uncertainty is therefore central to interpretable and reliable multimodal learning. We introduce conformal Shapley intervals, a framework that combines Shapley values with conformal inference to construct uncertainty-aware importance intervals for each modality. Building on these intervals, we propose a modality selection procedure with a provable optimality guarantee: conditional on the observed features, the selected subset of modalities achieves performance close to that of the optimal subset. We demonstrate the effectiveness of our approach on multiple datasets, showing that it provides meaningful uncertainty quantification and strong predictive performance while relying on only a small number of informative modalities.