A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation

TL;DR

This paper introduces a training-free, Gaussian Discriminant Analysis-based method for CLIP adaptation that outperforms or matches state-of-the-art techniques across various tasks without additional training.

Contribution

It revisits classical GDA and integrates it with CLIP's zero-shot classifier, enabling effective downstream classification without training or extra resources.

Findings

Outperforms state-of-the-art on 17 datasets

Effective in few-shot, imbalanced, and OOD tasks

Extends to base-to-new generalization and unsupervised learning

Abstract

Contrastive Language-Image Pretraining (CLIP) has gained popularity for its remarkable zero-shot capacity. Recent research has focused on developing efficient fine-tuning methods, such as prompt learning and adapter, to enhance CLIP's performance in downstream tasks. However, these methods still require additional training time and computational resources, which is undesirable for devices with limited resources. In this paper, we revisit a classical algorithm, Gaussian Discriminant Analysis (GDA), and apply it to the downstream classification of CLIP. Typically, GDA assumes that features of each class follow Gaussian distributions with identical covariance. By leveraging Bayes' formula, the classifier can be expressed in terms of the class means and covariance, which can be estimated from the data without the need for training. To integrate knowledge from both visual and textual modalities, we ensemble it with the original zero-shot classifier within CLIP. Extensive results on 17 datasets validate that our method surpasses or achieves comparable results with state-of-the-art methods on few-shot classification, imbalanced learning, and out-of-distribution generalization. In addition, we extend our method to base-to-new generalization and unsupervised learning, once again demonstrating its superiority over competing approaches. Our code is publicly available at \url{https://github.com/mrflogs/ICLR24}.