Efficient Multimodal Fusion via Interactive Prompting

TL;DR

This paper introduces PMF, an efficient multimodal fusion method that reduces training memory and parameters by using prompt-based interactions in unimodal transformers, achieving comparable performance with less resource usage.

Contribution

The paper proposes a novel prompt-based multimodal fusion framework that enhances flexibility and efficiency by adding prompts only in deep layers of unimodal transformers.

Findings

Achieves comparable performance to existing methods

Uses less than 3% trainable parameters

Saves up to 66% training memory

Abstract

Large-scale pre-training has brought unimodal fields such as computer vision and natural language processing to a new era. Following this trend, the size of multi-modal learning models constantly increases, leading to an urgent need to reduce the massive computational cost of finetuning these models for downstream tasks. In this paper, we propose an efficient and flexible multimodal fusion method, namely PMF, tailored for fusing unimodally pre-trained transformers. Specifically, we first present a modular multimodal fusion framework that exhibits high flexibility and facilitates mutual interactions among different modalities. In addition, we disentangle vanilla prompts into three types in order to learn different optimizing objectives for multimodal learning. It is also worth noting that we propose to add prompt vectors only on the deep layers of the unimodal transformers, thus significantly reducing the training memory usage. Experiment results show that our proposed method achieves comparable performance to several other multimodal finetuning methods with less than 3% trainable parameters and up to 66% saving of training memory usage.