DPL: Decoupled Prototype Learning for Enhancing Robustness of Vision-Language Transformers to Missing Modalities

TL;DR

This paper introduces Decoupled Prototype Learning (DPL), a novel prediction head architecture for vision-language transformers that improves robustness to missing modalities by adaptively selecting class prototypes based on available input information.

Contribution

DPL is a new architecture that explicitly adjusts decision processes for missing modalities, outperforming existing methods on multiple multimodal datasets.

Findings

DPL significantly improves robustness to missing modalities.

DPL outperforms state-of-the-art methods on several datasets.

DPL maintains compatibility with existing prompt-based frameworks.

Abstract

The performance of Visio-Language Transformers drops sharply when an input modality (e.g., image) is missing, because the model is forced to make predictions using incomplete information. Existing missing-aware prompt methods help reduce this degradation, but they still rely on conventional prediction heads (e.g., a Fully-Connected layer) that compute class scores in the same way regardless of which modality is present or absent. We introduce Decoupled Prototype Learning (DPL), a new prediction head architecture that explicitly adjusts its decision process to the observed input modalities. For each class, DPL selects a set of prototypes specific to the current missing-modality cases (image-missing, text-missing, or mixed-missing). Each prototype is then decomposed into image-specific and text-specific components, enabling the head to make decisions that depend on the information actually present. This adaptive design allows DPL to handle inputs with missing modalities more effectively while remaining fully compatible with existing prompt-based frameworks. Extensive experiments on MM-IMDb, UPMC Food-101, and Hateful Memes demonstrate that DPL outperforms state-of-the-art approaches across all widely used multimodal imag-text datasets and various missing cases.