A Retrospect to Multi-prompt Learning across Vision and Language

TL;DR

This paper explores multi-prompt learning in vision-language models, proposing an energy-based method to generate multiple prompts that improve adaptation and generalization across tasks and domains.

Contribution

It introduces EMPL, a novel energy-based multi-prompt learning approach, and provides a theoretical and empirical analysis of multi-prompt transfer benefits.

Findings

EMPL is parameter-efficient and enhances open-vocabulary generalization.

Multi-prompt augmentation outperforms single-prompt methods.

Theoretical analysis supports the superiority of multi-prompt learning.

Abstract

The vision community is undergoing the unprecedented progress with the emergence of Vision-Language Pretraining Models (VLMs). Prompt learning plays as the holy grail of accessing VLMs since it enables their fast adaptation to downstream tasks with limited resources. Whereas existing researches milling around single-prompt paradigms, rarely investigate the technical potential behind their multi-prompt learning counterparts. This paper aims to provide a principled retrospect for vision-language multi-prompt learning. We extend the recent constant modality gap phenomenon to learnable prompts and then, justify the superiority of vision-language transfer with multi-prompt augmentation, empirically and theoretically. In terms of this observation, we propose an Energy-based Multi-prompt Learning (EMPL) to generate multiple prompt embeddings by drawing instances from an energy-based distribution, which is implicitly defined by VLMs. So our EMPL is not only parameter-efficient but also rigorously lead to the balance between in-domain and out-of-domain open-vocabulary generalization. Comprehensive experiments have been conducted to justify our claims and the excellence of EMPL.