Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learning

TL;DR

This paper introduces RAGPT, a retrieval-augmented dynamic prompt tuning framework that significantly improves multimodal learning robustness with incomplete data by generating context-aware prompts through retrieval and missing information recovery.

Contribution

The paper proposes a novel retrieval-augmented dynamic prompt tuning method that addresses limitations of static prompts and dummy imputation in incomplete multimodal learning.

Findings

RAGPT outperforms baselines on three real-world datasets.

Dynamic prompts enhance robustness against missing modalities.

Retrieval-based missing information recovery improves task performance.

Abstract

Multimodal learning with incomplete modality is practical and challenging. Recently, researchers have focused on enhancing the robustness of pre-trained MultiModal Transformers (MMTs) under missing modality conditions by applying learnable prompts. However, these prompt-based methods face several limitations: (1) incomplete modalities provide restricted modal cues for task-specific inference, (2) dummy imputation for missing content causes information loss and introduces noise, and (3) static prompts are instance-agnostic, offering limited knowledge for instances with various missing conditions. To address these issues, we propose RAGPT, a novel Retrieval-AuGmented dynamic Prompt Tuning framework. RAGPT comprises three modules: (I) the multi-channel retriever, which identifies similar instances through a within-modality retrieval strategy, (II) the missing modality generator, which recovers missing information using retrieved contexts, and (III) the context-aware prompter, which captures contextual knowledge from relevant instances and generates dynamic prompts to largely enhance the MMT's robustness. Extensive experiments conducted on three real-world datasets show that RAGPT consistently outperforms all competitive baselines in handling incomplete modality problems. The code of our work and prompt-based baselines is available at https://github.com/Jian-Lang/RAGPT.