CLIP-Lite: Information Efficient Visual Representation Learning with Language Supervision

TL;DR

CLIP-Lite introduces an efficient contrastive learning method that reduces data and batch size requirements while outperforming CLIP in various visual recognition tasks by leveraging an information-theoretic approach with language supervision.

Contribution

It presents a novel, data-efficient contrastive learning framework that requires only one negative sample per positive, improving performance over CLIP with less data and computational resources.

Findings

+14.0% mAP on Pascal VOC classification

+22.1% top-1 accuracy on ImageNet

Superior performance in image/text retrieval and zero-shot tasks

Abstract

We propose CLIP-Lite, an information efficient method for visual representation learning by feature alignment with textual annotations. Compared to the previously proposed CLIP model, CLIP-Lite requires only one negative image-text sample pair for every positive image-text sample during the optimization of its contrastive learning objective. We accomplish this by taking advantage of an information efficient lower-bound to maximize the mutual information between the two input modalities. This allows CLIP-Lite to be trained with significantly reduced amounts of data and batch sizes while obtaining better performance than CLIP at the same scale. We evaluate CLIP-Lite by pretraining on the COCO-Captions dataset and testing transfer learning to other datasets. CLIP-Lite obtains a +14.0% mAP absolute gain in performance on Pascal VOC classification, and a +22.1% top-1 accuracy gain on ImageNet, while being comparable or superior to other, more complex, text-supervised models. CLIP-Lite is also superior to CLIP on image and text retrieval, zero-shot classification, and visual grounding. Finally, we show that CLIP-Lite can leverage language semantics to encourage bias-free visual representations that can be used in downstream tasks. Implementation: https://github.com/4m4n5/CLIP-Lite