Conditional Prompt Tuning for Multimodal Fusion

TL;DR

This paper introduces a novel conditional prompt tuning method for multimodal fusion that leverages one modality to guide the prompting of another, enabling efficient transfer of pretrained unimodal knowledge and achieving state-of-the-art results with minimal trainable parameters.

Contribution

It proposes a mixture of prompt experts (MoPE) framework for dynamic, instance-wise prompt routing, enhancing expressiveness and modularity in multimodal prompt tuning.

Findings

Achieves state-of-the-art results on three multimodal datasets.

Requires only 0.7% of trainable parameters compared to fine-tuning.

Demonstrates architecture-agnostic prompt tuning with improved scalability.

Abstract

We show that the representation of one modality can effectively guide the prompting of another modality for parameter-efficient multimodal fusion. Specifically, we first encode one modality and use its representation as a prior to conditionally prompt all frozen layers of the other modality. This is achieved by disentangling the vanilla prompt vectors into three types of specialized prompts that adaptively capture global-level and instance-level features. To better produce the instance-wise prompt, we introduce the mixture of prompt experts (MoPE) to dynamically route each instance to the most suitable prompt experts for encoding. We further study a regularization term to avoid degenerated prompt expert routing. Thanks to our design, our method can effectively transfer the pretrained knowledge in unimodal encoders for downstream multimodal tasks. Compared with vanilla prompting, we show that our MoPE-based conditional prompting is more expressive, thereby scales better with training data and the total number of prompts. We also demonstrate that our prompt tuning is architecture-agnostic, thereby offering high modularity. Extensive experiments over three multimodal datasets demonstrate state-of-the-art results, matching or surpassing the performance achieved through fine-tuning, while only necessitating 0.7% of the trainable parameters. Code will be released: https://github.com/songrise/ConditionalPrompt.