CLIP with Quality Captions: A Strong Pretraining for Vision Tasks

TL;DR

Enhancing caption quality in image-text datasets significantly improves CLIP's visual representations, leading to superior performance on dense prediction tasks and increased data efficiency, surpassing recent pretraining methods.

Contribution

This work demonstrates that high-quality captions in pretraining datasets enhance CLIP's effectiveness on dense vision tasks, outperforming recent state-of-the-art pretraining approaches.

Findings

CLIP with quality captions surpasses recent pretraining methods on dense tasks.

Improved caption quality yields 12.1% higher mIoU and 11.5% lower RMSE.

Mobile architectures benefit similarly from CLIP pretraining, achieving competitive results.

Abstract

CLIP models perform remarkably well on zero-shot classification and retrieval tasks. But recent studies have shown that learnt representations in CLIP are not well suited for dense prediction tasks like object detection, semantic segmentation or depth estimation. More recently, multi-stage training methods for CLIP models was introduced to mitigate the weak performance of CLIP on downstream tasks. In this work, we find that simply improving the quality of captions in image-text datasets improves the quality of CLIP's visual representations, resulting in significant improvement on downstream dense prediction vision tasks. In fact, we find that CLIP pretraining with good quality captions can surpass recent supervised, self-supervised and weakly supervised pretraining methods. We show that when CLIP model with ViT-B/16 as image encoder is trained on well aligned image-text pairs it obtains 12.1% higher mIoU and 11.5% lower RMSE on semantic segmentation and depth estimation tasks over recent state-of-the-art Masked Image Modeling (MIM) pretraining methods like Masked Autoencoder (MAE). We find that mobile architectures also benefit significantly from CLIP pretraining. A recent mobile vision architecture, MCi2, with CLIP pretraining obtains similar performance as Swin-L, pretrained on ImageNet-22k for semantic segmentation task while being 6.1$\times$ smaller. Moreover, we show that improving caption quality results in $10\times$ data efficiency when finetuning for dense prediction tasks.