Robust Multimodal Learning via Cross-Modal Proxy Tokens

TL;DR

This paper introduces cross-modal proxy tokens (CMPTs), a novel approach that improves the robustness of multimodal models to missing modalities by approximating absent class tokens through attention mechanisms, without extra modality generation.

Contribution

The paper proposes CMPTs, a simple and efficient method that enhances multimodal model robustness to missing data without requiring auxiliary networks or explicit modality generation.

Findings

Outperforms state-of-the-art baselines across multiple datasets.

Maintains strong performance with all modalities available.

Efficiently handles various missing rates with minimal computational overhead.

Abstract

Multimodal models often experience a significant performance drop when one or more modalities are missing during inference. To address this challenge, we propose a simple yet effective approach that enhances robustness to missing modalities while maintaining strong performance when all modalities are available. Our method introduces cross-modal proxy tokens (CMPTs), which approximate the class token of a missing modality by attending only to the tokens of the available modality without requiring explicit modality generation or auxiliary networks. To efficiently learn these approximations with minimal computational overhead, we employ low-rank adapters in frozen unimodal encoders and jointly optimize an alignment loss with a task-specific loss. Extensive experiments on five multimodal datasets show that our method outperforms state-of-the-art baselines across various missing rates while achieving competitive results in complete-modality settings. Overall, our method offers a flexible and efficient solution for robust multimodal learning. The code for this paper is available at: https://github.com/CSIPlab/Cross-Modal-Proxy-Tokens.