Deep Correlated Prompting for Visual Recognition with Missing Modalities

TL;DR

This paper introduces a novel prompt learning approach that leverages correlations between prompts and input features to improve multimodal visual recognition performance when some modalities are missing, addressing real-world data collection challenges.

Contribution

It proposes a correlation-aware prompt learning method that effectively handles missing modalities by exploiting inter-prompt and layer relationships, enhancing robustness of pretrained multimodal models.

Findings

Outperforms previous methods across multiple datasets and missing scenarios.

Demonstrates robustness across different modality-missing ratios and types.

Shows generalizability and reliability through extensive ablation studies.

Abstract

Large-scale multimodal models have shown excellent performance over a series of tasks powered by the large corpus of paired multimodal training data. Generally, they are always assumed to receive modality-complete inputs. However, this simple assumption may not always hold in the real world due to privacy constraints or collection difficulty, where models pretrained on modality-complete data easily demonstrate degraded performance on missing-modality cases. To handle this issue, we refer to prompt learning to adapt large pretrained multimodal models to handle missing-modality scenarios by regarding different missing cases as different types of input. Instead of only prepending independent prompts to the intermediate layers, we present to leverage the correlations between prompts and input features and excavate the relationships between different layers of prompts to carefully design the instructions. We also incorporate the complementary semantics of different modalities to guide the prompting design for each modality. Extensive experiments on three commonly-used datasets consistently demonstrate the superiority of our method compared to the previous approaches upon different missing scenarios. Plentiful ablations are further given to show the generalizability and reliability of our method upon different modality-missing ratios and types.