Are Multimodal Transformers Robust to Missing Modality?

TL;DR

This paper investigates whether Transformer models are inherently robust to missing modalities in multimodal data and finds that their robustness heavily depends on the fusion strategy, which varies by dataset.

Contribution

It is the first comprehensive study on Transformer robustness to missing modalities and introduces an automatic method to optimize fusion strategies for improved robustness.

Findings

Transformers are sensitive to missing modalities.

Optimal fusion strategies are dataset-dependent.

Proposed method improves robustness across benchmarks.

Abstract

Multimodal data collected from the real world are often imperfect due to missing modalities. Therefore multimodal models that are robust against modal-incomplete data are highly preferred. Recently, Transformer models have shown great success in processing multimodal data. However, existing work has been limited to either architecture designs or pre-training strategies; whether Transformer models are naturally robust against missing-modal data has rarely been investigated. In this paper, we present the first-of-its-kind work to comprehensively investigate the behavior of Transformers in the presence of modal-incomplete data. Unsurprising, we find Transformer models are sensitive to missing modalities while different modal fusion strategies will significantly affect the robustness. What surprised us is that the optimal fusion strategy is dataset dependent even for the same Transformer model; there does not exist a universal strategy that works in general cases. Based on these findings, we propose a principle method to improve the robustness of Transformer models by automatically searching for an optimal fusion strategy regarding input data. Experimental validations on three benchmarks support the superior performance of the proposed method.