Greedy Modality Selection via Approximate Submodular Maximization

TL;DR

This paper introduces a theoretical framework and efficient algorithms for selecting the most informative modalities in multimodal learning, balancing performance and computational constraints, with demonstrated effectiveness on synthetic and real datasets.

Contribution

It formulates a utility-based optimization approach for modality selection and develops algorithms leveraging approximate submodularity, connecting to Shapley-value scores.

Findings

Algorithms outperform baseline methods in modality selection

Effective on synthetic and real-world datasets

Reduces computational cost while maintaining performance

Abstract

Multimodal learning considers learning from multi-modality data, aiming to fuse heterogeneous sources of information. However, it is not always feasible to leverage all available modalities due to memory constraints. Further, training on all the modalities may be inefficient when redundant information exists within data, such as different subsets of modalities providing similar performance. In light of these challenges, we study modality selection, intending to efficiently select the most informative and complementary modalities under certain computational constraints. We formulate a theoretical framework for optimizing modality selection in multimodal learning and introduce a utility measure to quantify the benefit of selecting a modality. For this optimization problem, we present efficient algorithms when the utility measure exhibits monotonicity and approximate submodularity. We also connect the utility measure with existing Shapley-value-based feature importance scores. Last, we demonstrate the efficacy of our algorithm on synthetic (Patch-MNIST) and two real-world (PEMS-SF, CMU-MOSI) datasets.