PRE: Vision-Language Prompt Learning with Reparameterization Encoder

TL;DR

PRE introduces a reparameterization encoder for vision-language prompt learning, significantly improving generalization to unseen classes in zero-shot transfer tasks while maintaining efficiency and performance.

Contribution

It proposes a novel reparameterization encoder that enhances prompt generalization to unseen classes in vision-language models, addressing limitations of previous prompt learning methods.

Findings

Achieves 5.60% higher accuracy on new classes in 16-shot setting.

Improves harmonic mean by 3% over CoOp.

Demonstrates efficiency across 8 benchmarks.

Abstract

Large pre-trained vision-language models such as CLIP have demonstrated great potential in zero-shot transferability to downstream tasks. However, to attain optimal performance, the manual selection of prompts is necessary to improve alignment between the downstream image distribution and the textual class descriptions. This manual prompt engineering is the major challenge for deploying such models in practice since it requires domain expertise and is extremely time-consuming. To avoid non-trivial prompt engineering, recent work Context Optimization (CoOp) introduced the concept of prompt learning to the vision domain using learnable textual tokens. While CoOp can achieve substantial improvements over manual prompts, its learned context is worse generalizable to wider unseen classes within the same dataset. In this work, we present Prompt Learning with Reparameterization Encoder (PRE) - a simple and efficient method that enhances the generalization ability of the learnable prompt to unseen classes while maintaining the capacity to learn Base classes. Instead of directly optimizing the prompts, PRE employs a prompt encoder to reparameterize the input prompt embeddings, enhancing the exploration of task-specific knowledge from few-shot samples. Experiments and extensive ablation studies on 8 benchmarks demonstrate that our approach is an efficient method for prompt learning. Specifically, PRE achieves a notable enhancement of 5.60% in average accuracy on New classes and 3% in Harmonic mean compared to CoOp in the 16-shot setting, all achieved within a good training time.