CLIP's Visual Embedding Projector is a Few-shot Cornucopia

TL;DR

ProLIP is a simple, architecture-agnostic method that fine-tunes vision encoders for few-shot classification, achieving state-of-the-art results and excelling in transfer and adaptation tasks.

Contribution

It introduces ProLIP, a novel regularization-based fine-tuning approach for contrastively pretrained models, and proposes RLA, a hyperparameter-free linear adapter.

Findings

State-of-the-art performance on 11 few-shot benchmarks

Superior in transfer, domain generalization, and test-time adaptation

Faster training compared to prompt tuning

Abstract

We introduce ProLIP, a simple and architecture-agnostic method for adapting contrastively pretrained vision-language models, such as CLIP, to few-shot classification. ProLIP fine-tunes the vision encoder's projection matrix with Frobenius norm regularization on its deviation from the pretrained weights. It achieves state-of-the-art performance on 11 few-shot classification benchmarks under both ``few-shot validation'' and ``validation-free'' settings. Moreover, by rethinking the non-linear CLIP-Adapter through ProLIP's lens, we design a Regularized Linear Adapter (RLA) that performs better, requires no hyperparameter tuning, is less sensitive to learning rate values, and offers an alternative to ProLIP in black-box scenarios where model weights are inaccessible. Beyond few-shot classification, ProLIP excels in cross-dataset transfer, domain generalization, base-to-new class generalization, and test-time adaptation--where it outperforms prompt tuning while being an order of magnitude faster to train. Code is available at https://github.com/astra-vision/ProLIP .