Accelerating Conditional Prompt Learning via Masked Image Modeling for Vision-Language Models

TL;DR

ProMIM is a lightweight framework that improves prompt learning in vision-language models by integrating masked image modeling, enhancing robustness and generalization to unseen classes without significant computational overhead.

Contribution

We propose ProMIM, a plug-and-play masked image modeling approach that enhances existing prompt learning methods like CoOp and CoCoOp for better generalization.

Findings

ProMIM consistently improves zero-shot classification accuracy.

ProMIM enhances robustness against overfitting in few-shot learning.

ProMIM adds negligible computational cost to existing models.

Abstract

Vision-language models (VLMs) like CLIP excel in zero-shot learning but often require resource-intensive training to adapt to new tasks. Prompt learning techniques, such as CoOp and CoCoOp, offer efficient adaptation but tend to overfit to known classes, limiting generalization to unseen categories. We introduce ProMIM, a plug-and-play framework that enhances conditional prompt learning by integrating masked image modeling (MIM) into existing VLM pipelines. ProMIM leverages a simple yet effective masking strategy to generate robust, instance-conditioned prompts, seamlessly augmenting methods like CoOp and CoCoOp without altering their core architectures. By masking only visible image patches and using these representations to guide prompt generation, ProMIM improves feature robustness and mitigates overfitting, all while introducing negligible additional computational cost. Extensive experiments across zero-shot and few-shot classification tasks demonstrate that ProMIM consistently boosts generalization performance when plugged into existing approaches, providing a practical, lightweight solution for real-world vision-language applications.