EPIC: Efficient Prompt Interaction for Text-Image Classification

TL;DR

EPIC introduces a prompt-based interaction method that enhances multimodal text-image classification efficiency by reducing computational costs and parameters while maintaining or improving performance.

Contribution

The paper presents a novel prompt interaction strategy that significantly decreases resource consumption and parameters needed for multimodal classification tasks.

Findings

Reduces computational resources and trainable parameters by about 99%.

Achieves superior performance on UPMC-Food101 and SNLI-VE datasets.

Maintains comparable performance on MM-IMDB dataset.

Abstract

In recent years, large-scale pre-trained multimodal models (LMMs) generally emerge to integrate the vision and language modalities, achieving considerable success in multimodal tasks, such as text-image classification. The growing size of LMMs, however, results in a significant computational cost for fine-tuning these models for downstream tasks. Hence, prompt-based interaction strategy is studied to align modalities more efficiently. In this context, we propose a novel efficient prompt-based multimodal interaction strategy, namely Efficient Prompt Interaction for text-image Classification (EPIC). Specifically, we utilize temporal prompts on intermediate layers, and integrate different modalities with similarity-based prompt interaction, to leverage sufficient information exchange between modalities. Utilizing this approach, our method achieves reduced computational resource consumption and fewer trainable parameters (about 1\% of the foundation model) compared to other fine-tuning strategies. Furthermore, it demonstrates superior performance on the UPMC-Food101 and SNLI-VE datasets, while achieving comparable performance on the MM-IMDB dataset.