Prompt Distribution Learning

TL;DR

This paper introduces prompt distribution learning, a method that models prompt embeddings with a Gaussian distribution to improve adaptation of vision-language models for recognition tasks, especially with few samples.

Contribution

It proposes a novel approach that learns prompt output embeddings as a distribution, enabling better adaptation with limited data and outperforming existing methods.

Findings

Outperforms existing methods on 12 datasets

Achieves 9.1% relative improvement with one sample per category

Effectively models prompt diversity with Gaussian distribution

Abstract

We present prompt distribution learning for effectively adapting a pre-trained vision-language model to address downstream recognition tasks. Our method not only learns low-bias prompts from a few samples but also captures the distribution of diverse prompts to handle the varying visual representations. In this way, we provide high-quality task-related content for facilitating recognition. This prompt distribution learning is realized by an efficient approach that learns the output embeddings of prompts instead of the input embeddings. Thus, we can employ a Gaussian distribution to model them effectively and derive a surrogate loss for efficient training. Extensive experiments on 12 datasets demonstrate that our method consistently and significantly outperforms existing methods. For example, with 1 sample per category, it relatively improves the average result by 9.1% compared to human-crafted prompts.