MuAP: Multi-step Adaptive Prompt Learning for Vision-Language Model with Missing Modality

TL;DR

This paper introduces MuAP, a novel multi-step adaptive prompt learning framework designed for vision-language models with missing modalities, improving robustness and performance in incomplete data scenarios.

Contribution

It is the first to explore prompt learning with incomplete modalities and proposes a multi-step adaptive approach to generate and tune multimodal prompts.

Findings

MuAP significantly outperforms state-of-the-art methods on benchmark datasets.

The framework effectively mitigates modality imbalance issues.

Adaptive prompt tuning enhances model robustness with missing modalities.

Abstract

Recently, prompt learning has garnered considerable attention for its success in various Vision-Language (VL) tasks. However, existing prompt-based models are primarily focused on studying prompt generation and prompt strategies with complete modality settings, which does not accurately reflect real-world scenarios where partial modality information may be missing. In this paper, we present the first comprehensive investigation into prompt learning behavior when modalities are incomplete, revealing the high sensitivity of prompt-based models to missing modalities. To this end, we propose a novel Multi-step Adaptive Prompt Learning (MuAP) framework, aiming to generate multimodal prompts and perform multi-step prompt tuning, which adaptively learns knowledge by iteratively aligning modalities. Specifically, we generate multimodal prompts for each modality and devise prompt strategies to integrate them into the Transformer model. Subsequently, we sequentially perform prompt tuning from single-stage and alignment-stage, allowing each modality-prompt to be autonomously and adaptively learned, thereby mitigating the imbalance issue caused by only textual prompts that are learnable in previous works. Extensive experiments demonstrate the effectiveness of our MuAP and this model achieves significant improvements compared to the state-of-the-art on all benchmark datasets