Exploring Weak-to-Strong Generalization for CLIP-based Classification

TL;DR

This paper introduces class prototype learning (CPL), a method that improves CLIP-based classification by leveraging weak-to-strong generalization, especially effective with limited pretraining, achieving notable accuracy gains.

Contribution

The study extends weak-to-strong generalization to vision-language models and proposes CPL to enhance CLIP's classification through learned prototypes.

Findings

CPL improves classification accuracy by 3.67% over baselines.

Effective especially in limited pretraining scenarios.

Demonstrates robustness of weak supervision in multi-modal models.

Abstract

Aligning large-scale commercial models with user intent is crucial to preventing harmful outputs. Current methods rely on human supervision but become impractical as model complexity increases. When models surpass human knowledge, providing accurate feedback becomes challenging and inefficient. A novel solution proposed recently is using a weaker model to supervise a stronger model. This concept leverages the ability of weaker models to perform evaluations, thereby reducing the workload on human supervisors. Previous work has shown the effectiveness of weak-to-strong generalization in the context of language-only models. Extending this concept to vision-language models leverages these insights, adapting the proven benefits to a multi-modal context. In our study, we explore weak-to-strong generalization for CLIP-based classification. We propose a method, class prototype learning (CPL), which aims to enhance the classification capabilities of the CLIP model, by learning more representative prototypes for each category. Our findings indicate that, despite using a simple loss function under weak supervision, CPL yields robust improvements in targeted scenarios, particularly when pretraining is limited. Extensive experiments demonstrate that our approach is effective under these settings, achieving a 3.67% improvement over strong baseline methods.