Learning Unseen Modality Interaction

TL;DR

This paper introduces a novel approach for multimodal learning that generalizes to unseen modality combinations during inference by projecting features into a common space and using pseudo-supervision to improve robustness.

Contribution

It proposes a new method that enables multimodal models to handle unseen modality combinations, addressing a key limitation of existing approaches.

Findings

Effective across diverse tasks and modalities

Improves generalization to unseen modality combinations

Reduces overfitting through pseudo-supervision

Abstract

Multimodal learning assumes all modality combinations of interest are available during training to learn cross-modal correspondences. In this paper, we challenge this modality-complete assumption for multimodal learning and instead strive for generalization to unseen modality combinations during inference. We pose the problem of unseen modality interaction and introduce a first solution. It exploits a module that projects the multidimensional features of different modalities into a common space with rich information preserved. This allows the information to be accumulated with a simple summation operation across available modalities. To reduce overfitting to less discriminative modality combinations during training, we further improve the model learning with pseudo-supervision indicating the reliability of a modality's prediction. We demonstrate that our approach is effective for diverse tasks and modalities by evaluating it for multimodal video classification, robot state regression, and multimedia retrieval. Project website: https://xiaobai1217.github.io/Unseen-Modality-Interaction/.