Synergistic Prompting for Robust Visual Recognition with Missing Modalities

TL;DR

This paper introduces Synergistic Prompting (SyP), a novel framework that enhances the robustness of multi-modal visual recognition models against missing data by using dynamic and synergistic prompts for flexible adaptation.

Contribution

The paper proposes a new SyP framework with dynamic adapters and a synergistic prompting strategy, improving robustness to missing modalities in visual recognition tasks.

Findings

Significant performance gains over existing methods.

Robustness across various missing data scenarios.

Validated through extensive experiments and ablation studies.

Abstract

Large-scale multi-modal models have demonstrated remarkable performance across various visual recognition tasks by leveraging extensive paired multi-modal training data. However, in real-world applications, the presence of missing or incomplete modality inputs often leads to significant performance degradation. Recent research has focused on prompt-based strategies to tackle this issue; however, existing methods are hindered by two major limitations: (1) static prompts lack the flexibility to adapt to varying missing-data conditions, and (2) basic prompt-tuning methods struggle to ensure reliable performance when critical modalities are missing.To address these challenges, we propose a novel Synergistic Prompting (SyP) framework for robust visual recognition with missing modalities. The proposed SyP introduces two key innovations: (I) a Dynamic Adapter, which computes adaptive scaling factors to dynamically generate prompts, replacing static parameters for flexible multi-modal adaptation, and (II) a Synergistic Prompting Strategy, which combines static and dynamic prompts to balance information across modalities, ensuring robust reasoning even when key modalities are missing. The proposed SyP achieves significant performance improvements over existing approaches across three widely-used visual recognition datasets, demonstrating robustness under diverse missing rates and conditions. Extensive experiments and ablation studies validate its effectiveness in handling missing modalities, highlighting its superior adaptability and reliability.